Subject: [imap5] More random issues

From: Timo Sirainen <[email-masked]>
Date: Sat, 2 Aug 2008 04:08:19 +0300

Some things I thought about while waiting for my plane:

  - If message sequences are removed, then EXPUNGEs will be replaced  
by VANISHED.
   --> Remove CLOSE command.
  - Remove CHECK command. No-one uses it.
  - Require ESEARCH, drop SEARCH reply
  - Remove FETCH macros (ALL, FULL, FAST)
  - FETCH: UID field should always be the first item
  - FETCH ENVELOPE: Is Bcc field ever non-NIL? Is In-Reply-To field  
useful? I'd think References would be more useful.
  - Remove FETCH RFC822, RFC822.HEADER, RFC822.TEXT
  - Drop FETCH BODY*.PEEK? Never set \Seen implicitly.
  - Drop CAPABILITY command? Instead send CAPABILITY replies whenever  
they change (banner, post-starttls, post-login).
  - IDLE: Don't require 30 min timeout? Some clients already rely on  
this.. (Dovecot v1.1 never timeouts idleing connections.)
  - What at it, update some 32bit numbers to 64bit numbers (e.g.  
literals for APPEND, FETCH BODY[*], quota values)
  - Require supporting keywords, although there probably should be a  
way to limit their number?
  - List wildcards are more powerful than necessary? Just allow % and  
* at the end of the pattern.
  - SEARCH UNANSWERED/etc. are the same as SEARCH NOT ANSWERED/etc. -  
remove?
  - Require sequence-sets to be normalized so that there are no  
duplicates and numbers are increasing
  - Do we want to allow servers to return replies in different order  
than requested? They could still process commands in parallel, but the  
replies would always be in the expected order. Few clients or servers  
support out-of-order processing..
  - Explicitly specify how SEARCH FROM/TO/CC should match the address.
  - Allow STATUS on selected mailbox? No real reason to disallow it..
  - Remove all special cases from INBOX: It's now case-sensitive and  
it's not in its own special namespace.
  - Do we keep subscriptions? If so, allow subscribed-state to be part  
of the mailbox, so that deleting a mailbox also unsubscribes from it.
  - I'd go and make namespaces work differently.. So that LIST * would  
list all mailboxes and namespace would simply describe which parts of  
the list are private/public/shared. Currently most clients ignore  
namespaces, and those that don't use it the way I described. This  
wouldn't allow specifying UW-IMAP style mailbox drivers, but I don't  
think users should be able to have that power anyway.

As for actually removing MSNs .. I agree that they can be useful, but  
then again they introduce a lot of complexity (e.g. most servers don't  
preserve messages until they're expunged in all session, causing  
problems when FETCHing epunged messages). Also I think most of the MSN  
usages can be worked around more or less efficiently with UIDs. And  
keeping track of MSNs properly does seem to be difficult for both  
clients and servers. So all in all everything would be a lot simpler  
and more robust if we just got rid of them.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Fri, 1 Aug 2008 19:29:47 -0700

> --> Remove CLOSE command.

If a client does not need to synchronize (e.g., a kiosk), why should it be
required to accept synchronization?

> - Remove CHECK command. No-one uses it.

It is used, and is necessary if legacy mail stores are supported.  It can be
removed if IMAP5 defines a mail store (which in turn will shrink its
applicable audience).

> - FETCH: UID field should always be the first item

Probably OK.

> useful? I'd think References would be more useful.

Yes (but not very often) and yes.

Rather than attacking ENVELOPE in this way, it would be better to
have some good way for the client to define what it wants from a
parse of the message.

> - Drop FETCH BODY*.PEEK? Never set \Seen implicitly.

I don't think that adding mandatory RTTs is a good idea.

> they change (banner, post-starttls, post-login).

Good idea IMHO.

> this.. (Dovecot v1.1 never timeouts idleing connections.)

This would be a great way to convince sites not to run IMAP5 servers:
"if you run an IMAP5 server, you end up with an unbounded number
of sessions that never go away".

> literals for APPEND, FETCH BODY[*], quota values)

Good idea, although I personally shudder at>4GB messages.

> - SEARCH UNANSWERED/etc. are the same as SEARCH NOT ANSWERED/etc. -

NOT was not in IMAP2.  I agree that NOT means that the UNxxx forms
can be removed.

> duplicates and numbers are increasing

That changes the behavior of n:* for a UID set where n is <= UIDLAST
but is> the UID for message *.

> support out-of-order processing..

The existing IMAP already specified that responses can be returned out
of order.  IMHO, requiring responses to be in order defeats most of the
point of multiple commands in progress.

> - Allow STATUS on selected mailbox? No real reason to disallow it..

Besides the fact that it's a stupid thing to do and it indicates a poorly-
written client that doesn't implement IMAP state?

> it's not in its own special namespace.

It'll be quite amusing to watch the (to me, highly-predictable) outcome
of that decision.  I'll be laughing my head off.

If you don't see what I'm talking about, I suggest that you take the time
to think it through.

> problems when FETCHing epunged messages).

That's a bad implementation in the servers.

By approving such behavior, you're essentially saying that it is alright
to pull the rug out from under clients.

> Also I think most of the MSN

There is a pre-existing example of a UID-only messaging protocol:
NNTP.  That characteristic of NNTP has been shown to have problems
that are exacerbated as the number of messages becomes large.

The advocates of UID only are, in effect, assuming that all clients
synchronize with a local state that is preserved across sessions.  Think
about what happens when that is not the case.

Yes, "most IMAP clients suck".  Among other reasons, most have a
UID-only mindset.

_________________________________________________________________
Use video conversation to talk face-to-face with Windows Live Messenger.
http://www.windowslive.com/messenger/connect_your_way.html?ocid=TXT_TAGLM_WL_Refresh_messenger_video_072008
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Philip Guenther <[email-masked]>
Date: Sat, 2 Aug 2008 11:50:27 -0600

On Fri, 1 Aug 2008, Mark Crispin wrote:
> It is used, and is necessary if legacy mail stores are supported.  It 
> can be removed if IMAP5 defines a mail store (which in turn will shrink 
> its applicable audience).

Years ago, a friend and I were discussing IMAP stuff and CHECK came up.  
We came to the conclusion that the spec should have included text that ran 
something like:
	An IMAP server MAY delay saving to permanent storage changes to 
	message flags.  However, all flag changes MUST be recorded before 
	the server sends an OK response to any of the following commands: 
	CHECK, EXPUNGE, LOGOUT, or an APPEND or COPY command that names 
	the currently selected mailbox.  Before sending an OK or NO 
	response to the LOGOUT, EXAMINE, SELECT, or CLOSE commands, the
	server MUST try to record all flags changes; if unable to do so it 
	MUST send an untagged NO response with the [LOSTCHANGES] response 
	code.

I.e., CHECK would serve as a "group commit" operation for flag changes and 
server can delay saving flag data until the client tells them to.  Note 
that servers would be permitted to silently lose changes if the client 
drops the connection or times out, though I suspect 'market pressure' 
would prevent servers from doing that willingly.

Philip Guenther

(Regarding extensions: UNSELECT should behave like CLOSE; annotations 
should have the same delayed commit semantics as flags; with CONDSTORE, a 
given change must be saved before sending the FETCH response containing 
both it and a MODSEQ data item.)
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Sat, 2 Aug 2008 11:08:04 -0700

> MUST send an untagged NO response with the [LOSTCHANGES] response

I think that this is a good idea.

I also believe that if IMAP5 fails to recognize the possible need for "group
commit" then one of two things will happen:

[1] IMAP5 will fail to gain acceptance due to being tied to a very narrow
data store.

[2] Servers which do group commit will happen anyway, and there will be
no way for clients to know or control the behavior.

Many data stores (including the cloud based data stores) do NOT commit
as part of write, and instead cache multiple writes for later commit.

I also note the distressing tendency of some people to think that all the
message store world is, or should be, maildir/mh/newspool.

_________________________________________________________________
Got Game? Win Prizes in the Windows Live Hotmail Mobile Summer Games Trivia Contest
http://www.gowindowslive.com/summergames?ocid=TXT_TAGHM
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Sat, 2 Aug 2008 23:41:58 +0300

> no way for clients to know or control the behavior.

What clients issue CHECK currently and when? I've no idea how I would  
recommend clients to handle them. Should they send it once every n  
minutes? Should there be a command to explicitly do that (how would  
user know about that, and how many would actually use it)? Why should  
a user or a client even care about when this is done by the server?

---

From: Mark Crispin <[email-masked]>
Date: Sat, 2 Aug 2008 14:09:40 -0700

> a user or a client even care about when this is done by the server?

Pine and Alpine upgrades its routine polling when idle from NOOP to CHECK
every 10 minutes or so.  The user can also force a poll, and if the user does a
few consecutive forced polls that gets upgraded to a CHECK.

In some mail stores, CHECK re-reads all flags in all messages, and sends a
FETCH FLAGS update for the messages which were previously read but now
the flags have changed.  This is the only way to guarantee flag synchronization
with those stores, since there is no automatic means of detecting flag changes
and the flag sweep is a potentially costly operation.

In an ideal mail store, CHECK is no different than NOOP.  In a less than ideal
world where CHECK does something, a server should automatically do the
CHECK operation at some time interval even if the client does not.

I think that the issue boils down to this:

If the group wants to define the semantics of an ideal mail store, then CHECK
can go away.

If the group wants to allow IMAP5 to export any legacy store, it must export
all commonly-used legacy stores, not just maildir.

It is unreasonable to remove support for some legacy stores that are still in
common use, yet keep support for other legacy stores.

My mischievous message was to make a point.

_________________________________________________________________
Reveal your inner athlete and share it with friends on Windows Live.
http://revealyourinnerathlete.windowslive.com?locale=en-us&ocid=TXT_TAGLM_WLYIA_whichathlete_us
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Randall Gellens <[email-masked]>
Date: Sat, 2 Aug 2008 16:51:49 -0700

>   - FETCH: UID field should always be the first item

Sounds reasonable, although I might be missing something.

> useful? I'd think References would be more useful.

Bcc can exist on messages you sent, as well as on messages received 
via Bcc generated by some clients,

>   - Drop CAPABILITY command? Instead send CAPABILITY replies 
> whenever they change (banner, post-starttls, post-login).

Seems worth considering.

> this.. (Dovecot v1.1 never timeouts idleing connections.)

If there is a NAT or other middleware box between the client and the 
server, it may be necessary to refresh the connection periodically. 
The lemonade deployments document talked about it, how it's better if 
it isn't needed but it often is needed 
(draft-ietf-lemonade-deployments-09.txt).

> literals for APPEND, FETCH BODY[*], quota values)

Sounds appealing, but would this be too much of a break, making it 
more work for current implementations to support IMAP5?

Your additional suggestions and thoughts are helpful and I look 
forward to the discussion.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
All probabilities are 50%.  Either a thing will happen or it won't.
This is especially true when dealing with someone you're attracted to.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 4 Aug 2008 00:27:31 +0300

> few consecutive forced polls that gets upgraded to a CHECK.

If this is the preferred behavior, why couldn't the server do this  
internally when seeing NOOPs?

> and the flag sweep is a potentially costly operation.

Yes, Dovecot + mbox CHECK behaves identically to UW-IMAP + mbox CHECK,  
but I see very few reasons for a client to be able to force this flag  
writing. (About the only reason I can think of is for my imaptest  
tool..)

> all commonly-used legacy stores, not just maildir.

Even if an ideal mail store is defined, I think it would be possible  
to export all legacy mail stores with IMAP5. It just requires some  
more work and maybe some IMAP5-server specific metadata stored outside  
the lagacy store (similar to how Dovecot's index files are stored).  
The behavior may not be always optimal, but I think it would be  
possible in any case. None of the suggestions I've seen/written so far  
would prevent me from supporting mbox with Dovecot IMAP5.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Sun, 3 Aug 2008 19:11:08 -0400

On Aug 2, 2008, at 7:51 PM, Randall Gellens wrote:

> The lemonade deployments document talked about it, how it's better  
> if it isn't needed but it often is needed (draft-ietf-lemonade- 
> deployments-09.txt).

That's a different thing. Dovecot currently sends a "* OK Still here"  
every 2 minutes, so if the connection has died it'll know about it  
soon enough anyway (without forcing a client to send something every  
30 mins). Although mobile people don't like it since it increases the  
bandwidth usage..

> more work for current implementations to support IMAP5?

Hmm. Could be a bit too early for it. Also I remembered wrong that  
quota was defined in bytes, but since it's kilobytes 4 TB limit should  
be enough for a few more years.

---

From: Timo Sirainen <[email-masked]>
Date: Sun, 3 Aug 2008 19:26:12 -0400

> but is> the UID for message *.

What clients use UID FETCH n:* currently? I know Outlook does, but it  
doesn't really want to fetch the last message in that case (which is  
probably why Courier doesn't return it either..). Perhaps disallow *  
alone and redefine n:* to mean n:2^31-1 (or perhaps remove *  
completely).

> point of multiple commands in progress.

Perhaps, but is there much point in supporting that anyway? Most  
clients that want that just use multiple connections, which works with  
more servers.

> written client that doesn't implement IMAP state?

It's still an extra unnecessary restriction that takes space in the  
specification. One sentence recommending not to use it for selected  
mailbox should be enough (instead of a whole paragraph with a SHOULD  
NOT).

> If you don't see what I'm talking about, I suggest that you take the  

Assuming client UI handles it well, I don't really see what problems  
there could be. To me it seems INBOX is (or should be) just a normal  
mailbox which could be assumed to always exist.

> That's a bad implementation in the servers.

You always talk about making IMAP work with legacy mail stores, but  
ignore that this is the most difficult feature to implement for such  
mail stores (and not just for legacy mail stores, but for many modern  
ones as well not specifically designed for this use case).
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Sun, 3 Aug 2008 19:51:46 -0700

> doesn't really want to fetch the last message in that case

Knowing the current last message is important for any synchronization
algorithm that I'd use.  Losing n:* would mean more RTTs.

> (which is
> probably why Courier doesn't return it either..).

Courier is broken by design.  My IMAP client code has to have a special
switch identifying a Courier server, in order to do things in a far less
efficient way in order to avoid Courier's defects.

> Perhaps disallow *

If you do that, then you need MSNs to access the last message without an
extra RTT.

>> of order.

What, then, is the point of tagging commands/responses?

>> Besides the fact that it's a stupid thing to do and it indicates a
>> poorly-written client that doesn't implement IMAP state?

STATUS is potentially an extremely expensive operation involving locks
on some mail stores.

Is the goal to make IMAP5 unimplementable on most mail stores?  There
are enough complaints about the difficulty of implementing IMAP4 on
some mail stores.  Do you want to lock out the mail stores that are currently
implementable in IMAP4?

> mailbox which could be assumed to always exist.

Those who refuse to learn the lessons from the past are condemned to
repeat them.

> ones as well not specifically designed for this use case).

Nonsense.

The mail store that you have trouble with is maildir, which certainly can
not be called "modern".

Why don't you call IMAP5 MMAP for the Maildir Message Access
Protocol and be done with it?

---

From: Mark Crispin <[email-masked]>
Date: Sun, 3 Aug 2008 20:03:42 -0700

> internally when seeing NOOPs?

Because the client may need to have control over the commit event.

If you delete CHECK, there'll end up having to be an IMAP5
extension for commit.  Commit is particularly important in
distributed data stores where you can't assume atomic updates
across the cluster.

> to export all legacy mail stores with IMAP5.

I disagree.  Your fallacy is the presumption that there are not legacy
applications along with the legacy store and/or that it is sufficient to
rebuild the auxiliary file when it is detected that something else
mucked with the legacy store.

All it needs is one undetected change.  A size check alone isn't
enough; the chances of coincidence are startlingly high.

Even if you address those problems, then you get into what happens
when the conflicts are frequent, and the auxiliary file rebuilds cost
more than the file benefits.

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 4 Aug 2008 18:15:52 -0400

> algorithm that I'd use.  Losing n:* would mean more RTTs.

It could still be done with "SEARCH *" (and SEARCHRES to avoid extra  
RTT).

> What, then, is the point of tagging commands/responses?

Nothing I suppose. Let's get rid of them as well. :)

> STATUS is potentially an extremely expensive operation involving locks

But on the selected mailbox the server should already know all that  
information anyway and be able to return it directly.

> some mail stores.  Do you want to lock out the mail stores that are  

I'd mainly want to make the specification as easy to understand and as  
unambiguous as possible.

> The mail store that you have trouble with is maildir, which  

It's not just maildir. I'm aware of 3 ways to make this work:

1. mix/mbx style: Implement a shared-lock and remove the messages once  
there's only one reader. I don't really like this method, since it  
doesn't guarantee that the messages ever get removed.

2. Delayed expunges. All expunged messages are kept on the filesystem  
and they're deleted "later", typically at night when there's less work  
to be done. I don't think most small installations want to use this (I  
know I don't).

3. Mailbox format which keeps track of which sessions see which  
messages. This would actually work and I've figured out a way that  
would work, but it's quite a lot of extra work for something that is  
rarely actually needed, so I haven't bothered to implement it.

As far as I know UW-IMAP is the only server implementing 1 and the  
rest of the servers where it appears to work are using 2. Most of the  
servers implement none of them and just fail the FETCH in one way or  
another.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Mon, 4 Aug 2008 15:33:18 -0700

> It could still be done with "SEARCH *" (and SEARCHRES to avoid extra

It's still an additional RTT.  You're assuming that the only thing being
fetched is the UID.

> Nothing I suppose. Let's get rid of them as well. :)

At this point, you no longer have IMAP.  Call it something else.

> information anyway and be able to return it directly.

This argument betrays the same sort of thinking that led to the disasterous
POSIX locking semantics.

Please read the BSD man page on fctnl() locking and in particular the
paragraph starting with "This interface follows...".

>> some mail stores. Do you want to lock out the mail stores that are
>> currently implementable in IMAP4?
>

Deleting functionality that has been part of IMAP from inception, and
molding IMAP into a protocol to export specifically maildir, does not
achieve your stated goals.

> doesn't guarantee that the messages ever get removed.

Do you prefer that clients crash when a message vanishes without a proper
expunge announcement?

That happens today.

> to be done. I don't think most small installations want to use this (I

The presumption being made is that after about 30 minutes or so, only a
truly abusive client will fail to do an operation which negotiates the
expunge.

> rarely actually needed, so I haven't bothered to implement it.

Speak for yourself.  This is exactly what I intended to implement in the
next phase of mix.

> Most of the

Users of those servers just accept that every so often their client
will crash, and they're told that it's because of IMAP.

-- Mark --

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 4 Aug 2008 21:00:51 -0400

> It's still an additional RTT.  You're assuming that the only thing  

No, SEARCHRES allows pipelining commands and requires server to wait  
for the previous SEARCH to finish before processing $. So for example  
this can be sent all at once:

1 SEARCH *
2 FETCH $ (ENVELOPE BODY)

> achieve your stated goals.

I've been now mainly just throwing ideas of what could be removed. I  
don't have all that strong opinions yet what should be kept and what  
should be removed. And I'm not thinking about maildir specifically,  
but I do think a lot of server implementations use mailbox formats  
that could be thought of somewhat equivalent.

> Do you prefer that clients crash when a message vanishes without a  

No-one's yet complained to me about that happening. I guess it also  
depends on what exactly you mean by "vanishes". Dovecot's still  
returning the replies, but it just returns message body (and other  
fields it has lost) as NIL.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Mon, 4 Aug 2008 19:50:20 -0700

> I've been now mainly just throwing ideas of what could be removed.

That is neither a good nor appropriate way to proceed.

Identify core functionality first.

>> Do you prefer that clients crash when a message vanishes without a
>> proper expunge announcement?

I have heard from users who switched to my imapd, and thanking me because my imapd doesn't cause their clients to crash with a shared expunge.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 18 Aug 2008 09:06:53 +0100

> required to accept synchronization?

It doesn't need to request it in the first place, or can explicitly  
turn it off prior to unselecting the mailbox, by pipelining whatever  
command shuts up the server with the EXPUNGE/UNSELECT.

> > - FETCH: UID field should always be the first item

I'd be inclined to use the UID where we currently have the MSN. I   
could go for "* FETCH x ..." where x is the UID, too.

> I don't think that adding mandatory RTTs is a good idea.

I use the non-PEEK forms as a client dev, but I'd be happy in my   
guise as server developer to drop implicit Seen, it's a bit of an   
annoyance.

> of sessions that never go away".

Yes, but XMPP has this property too, and the major problem there is  
actually the invalid presence caused by the dangling session, and not  
the dangling session itself. This might be a difference in the amount  
of state per session, of course, but I think that drops if servers  
need no longer "pend" expunges and the like.

XMPP has, of course, one massive advantage in as much as there exists  
a mechanism for servers to ping clients - perhaps we should simply  
include such a mechanism, but I suspect that if we said...

	An untagged PING response MAY be sent at any time, and MUST cause  
the client to issue a command (such as NOOP) in a prompt manner.  
Servers MAY unilaterally close the connection of clients which do not  
respond within a certain timeout, this timeout MUST NOT be lower than  
5 minutes.

... then I suspect we'd have interop problems up to our eyeballs,  
with client developers missing this particular item. Making this  
timeout 30 minutes, however, would in principle make things no worse.

> Good idea, although I personally shudder at>4GB messages.

Yes... I'm not convinced that all 32-bit counters need updating, and   
I wonder what would happen if they were.

I'm pretty sure that we'll never need a 4G message.

> NOT was not in IMAP2.  I agree that NOT means that the UNxxx forms

I think we could probably have just SEARCH FLAGS (\Answered \Seen) or  
something, and simplify the whole thing.

> but is> the UID for message *.

Yes, but I think we could mandate it for the server-sent sets at  
least.

> point of multiple commands in progress.

Yeah, this really does do no harm. Especially if you remove the  
ambiguities by using tagged intermediates where sensible.

> poorly-

Ah, but it can tell you the UNSEEN count, whilst nothing else will  
unless you maintain total flag state.

In principle, STATUS on the SELECTed mailbox is equivalent to one or   
more SEARCH RETURN (COUNT) constructs anyway, so if that's possible,   
then I'd have thought STATUS should be - yes, it may require special   
handling, but that's the server's problem - I see nothing  
tremendously difficult about handling STATUS on the currently  
selected mailbox as a special case for those (rare) servers which  
have issues with it at a lower level.

> If you don't see what I'm talking about, I suggest that you take   

The various special cases are quite important. For example, a mailbox  
  rename of INBOX basically has to remain special in many   
implementations.

Even in our implementation, where INBOX is really not terribly  
special at all, the special cases are mostly useful, and I don't see  
they do any harm.

> That's a bad implementation in the servers.

This may be true. But a good protocol has the ability to be badly  
implemented, yet conformant, easily - I don't think this applies to   
IMAP, or else we'd see rather more conformant servers. Arguing that  
99% of servers are wrong seems equivalent to arguing that the design  
is wrong.

> to pull the rug out from under clients.

Oh, I think that's a given. But the world seems to be moving toward  
push-based architectures, and UIDs fit in this with much better -  
pulling the rug is fine as long as clients are expecting it and are  
clearly told.

Perhaps a better model would be to have clients able to "lock"  
messages from being expunged, so they could choose to stand more  
firmly on the rug, as it were - but I'm not all that bothered, as a  
client author. I find the rug is pulled very infrequently, and  
doesn't unduly affect me when it is.

> about what happens when that is not the case.

My client is specifically designed to handle particularly large   
mailboxes; it does so using the numeric, strictly increasing property  
of UIDs, but doesn't use MSN-as-ref at any point. It was originally   
written to not use a persistent cache at all, and it's still very  
much optional. I think this counts as thinking about it. :-)

I do agree that replacing UIDs with stuff based on SHA-1 (or whatever  
the flavour of the week is), won't scale well for offline  
synchronization, and this matches the message-id-as-ref of NNTP quite  
closely. That's not to say that having the SHA-1 is itself a problem,  
  just that it's nothing like as good as UIDs.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 18 Aug 2008 21:14:56 -0700 (PDT)

> I'm pretty sure that we'll never need a 4G message.

How do you plan to freeze the growth of multimedia content and Internet 
bandwidth?

--lyndon

   When in trouble or in doubt, run in circles; scream and shout.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 19 Aug 2008 07:23:20 +0300

> guise as server developer to drop implicit Seen, it's a bit of an   

I guess this is just a client UI issue, but I prefer the client to not
mark the message \Seen immediately when opening since I may have just
accidentally clicked it. So I thought more clients were already doing
that.

In any case dropping non-PEEK shouldn't add RTTs, you can just as well
pipeline the STORE commands (it shouldn't matter if you add \Seen flag
for messages that already had it).

> Yes, but I think we could mandate it for the server-sent sets at  

Isn't it already in the few places where the server even sends them?

> special at all, the special cases are mostly useful, and I don't see  

I'd prefer to just get rid of mailboxes and use only tags/keywords.
Although that still requires adding more powerful keyword handling (e.g.
"list all keyword counts") and maybe some kind of virtual views.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 18 Aug 2008 00:51:49 +0100

It doesn't need to request it in the first place, or can explicitly  
turn it off prior to unselecting the mailbox.

> have some good way for the client to define what it wants from a
Agreed.

> whenever
Agreed.

I think we can safely ditch the timeout, but certainly advertising  
what the timeout is, if any, would be useful.

In principle, STATUS on the SELECTed mailbox is equivalent to one or  
more SEARCH RETURN (COUNT) constructs anyway, so if that's possible,  
then I'd have thought STATUS should be - yes, it may require special  
handling, but that's the server's problem.

This may be true. But a good protocol has the ability to be badly  
implemented, yet conformant, easily - I don't think this applies to  
IMAP, or else we'd see rather more conformant servers - in  
particular, more than one.

Oh, I think that's a given. But the world seems to be moving toward  
lockless, push-based architectures, and UIDs fit in this with much  
better - pulling the rug is fine as long as clients are expecting it  
and are clearly told.

My client is specifically designed to handle particularly large  
mailboxes; it does so using the numeric, strictly increasing property  
of UIDs, but doesn't use MSN-as-ref at any point. It was originally  
written to not use a persistent cache at all. I think this counts as  
thinking about it. :-)

I do agree that replacing UIDs, too, with stuff based on SHA-1 (or  
whatever the flavour of the week is), won't scale well for offline  
synchronization, and this matches the message-id-as-ref of NNTP quite  
closely. That's not to say that having the SHA-1 is itself a problem,  
just that it's nothing like as good as UIDs.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 19 Aug 2008 21:25:27 +0200

Lyndon Nerenberg writes:
> How do you plan to freeze the growth of multimedia content and 
> Internet bandwidth?

I agree.

We ought to limit the size to 2G or allow more than 4G, and <2G is IMO 
not viable.

Arnt

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 22 Aug 2008 14:42:17 +1000

>   just that it's nothing like as good as UIDs.

The annoying thing with UIDs (as a server implementor) is that they must
appear in strictly increasing order.  This means that your lmtp delivery
agent can't grab an ID and run with it, it needs to hold a global lock on
the mailbox from the moment it first needs an ID until it has fully
committed the file with that UID to persistent storage.

Which is fine until you have a multi-master server get a split brain
and start allocating the same IDs to different messages.  The only way
to solve this is to re-id BOTH messages, and without some form of opaque
globally unique handle (like, for-example, a sha1) the clients at both
ends then wind up fetching the entire message again.

The cost of offline synchronization that you're talking of here is 
pretty much solved by MODSEQ, since you only have to keep the last
HIGHESTMODSEQ value for that mailbox and you have a small set of
IDs that actually need to be double-checked.

(in the split-brain scenario I outlined above, you would basically have
to find the ancestor where they split and re-MODSEQ anything past that
date, but at least that would just cause a slightly larger metadata
sync rather than a full message body resync)

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 07:58:55 +0300

On Fri, 2008-08-22 at 14:42 +1000, Bron Gondwana wrote:
> appear in strictly increasing order.

Unfortunately changing this would make it a lot more complex for client
to fetch newly seen messages. You'd also need some kind of a new
untagged reply that updates message sequences (similar to VANISHED):

* APPENDED 123:128,150

Also if it's done the client can't really order messages by
sequence/UID anymore, since there's no guarantees that they're
ascending.

An alternative would be to make sequence<->uid mapping completely
separate from each others so that the new messages would be given
appended sequences, but UIDs could be more or less random. But that
would really make things work in weird ways.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Philip Guenther <[email-masked]>
Date: Thu, 21 Aug 2008 23:13:02 -0600

On Fri, 22 Aug 2008, Bron Gondwana wrote:
...
> appear in strictly increasing order.
...
> IDs that actually need to be double-checked.

Doesn't the MODSEQ value have the exact same "strictly increasing" 
property as UIDs?  And "more so", as it applies to all metadata operations 
too and not just injection?  If you're considering tossing that property 
of UIDs, then CONDSTORE should be completely non-starter, no?

Philip Guenther

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 22 Aug 2008 15:35:24 +1000

> untagged reply that updates message sequences (similar to VANISHED):

Not really.  That's what MODSEQ is for.  I agree it would kind of suck
without MODSEQ.

The big problem I see now with IMAP4 is the alternative case - trying to
find a full list of what's changed since I last connected.  I.E. flag
updates.  This can't be done without fetching a line for every single
message in the mailbox.

> &#65279;Also if it's done the client can't really order messages by

Yeah, that's a little more sucky if they have exactly the same
INTERNALDATE.  You don't have quite such a stable sort order.
Mind you, I'd be happy to sort by something else within the same
exactly INTERNALDATE second (like the sha1 for example) - things
that arrive that close together really did happen in undefined order
unless they were done by the same client.

> would really make things work in weird ways.

INTERNALDATE plus client counter would be enough to get a guaranteed
sort order.  Sure things might appear slightly out-of-order with
multiple merge sources, but only by one second if your clocks are
accurate.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 22 Aug 2008 15:40:17 +1000

> of UIDs, then CONDSTORE should be completely non-starter, no?

I'm more happy to bump a MODSEQ up on merge, since it's a single int64
per merge, and it just means "you need to double-check your metadata
on this" rather than "this is a totally brand-new message so you need
to fetch the lot".

You don't expect a MODSEQ to never change, basically.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 08:42:02 +0300

> of UIDs, then CONDSTORE should be completely non-starter, no?

The difference is that modseqs change "all the time", while UIDs don't.
If a modseq changes unneededly the client needs to fetch maybe extra 30
bytes for each changed message. If an UID changes (due to a conflict
when syncing two servers), the client needs to fetch the entire message
body again, and potentially lose any locally stored user-given
annotations that were assigned to the specific UID. So the server could
easily get away with sometimes giving a new modseq for all messages, but
not for giving a new UID for all messages.

Bron also mentioned this being a problem with lmtpd (or I assume also
with COPY/APPEND), but I don't really see an issue there. Assuming a
good enough mail storage format, it's possible for the server to write
the message entirely to the mailbox and grab a lock only for the small
duration of assigning an UID (or you could possibly even get away with
not using a lock at all, but instead rely on link() or rename()
behavior).
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 08:48:52 +0300

> Not really.  That's what MODSEQ is for.  I agree it would kind of suck

Even with modseqs, think about what happens inside an IMAP session:

1. Initially you have one message: sequence 1 = UID 5
2. Then you'll get a new message with UID 3

What happens?

a) seq 1 = uid 5, seq 2 = uid 3?
b) seq 1 = uid 3, seq 2 = uid 5?

If b), you'll need the APPENDED untagged reply, otherwise the client's
sequence <-> uid mapping becomes completely broken.

If a), you can't anymore rely on sequence and UID ordering being the
same. And what will happen if the client reconnects? Must the mapping
still stay the same, or will they now be sorted based on UID and given
b) mapping? There are probably other issues with this that I can't think
of right now.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 22 Aug 2008 16:17:33 +1000

> b) seq 1 = uid 3, seq 2 = uid 5?

yes, one of those.  I would say it depends on the INTERNALDATE
myself.  Internaldate + client sequence number + global (sha1 for
example) id makes a totally stable sort that's guaranteed to keep
data from the same session in order while not imposing global
locking requirements.

> sequence <-> uid mapping becomes completely broken.

if you're fetching by id, sure.  Again - this is no different than
if a client had deleted a message out of the middle of your set.
The sequence <-> uid mapping is still bogus as soon as there's a
change on the server side.  If you're referring to stuff by SEQ then
you're in for a world of hurt in a shared mailbox environment
regardless.

> b) mapping? There are probably other issues with this that I can't think

The whole point is that you don't have an ordered UID any more.  I don't
even like the idea of the UID being a single number - because then you
need a global number source that's synchronised.  But other people seem
to like the single number UID because it's simple and cheap to store on
a basic client.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 22 Aug 2008 16:09:14 +1000

> not for giving a new UID for all messages.

Yes, that.

> not using a lock at all, but instead rely on link() or rename()

It's: 

a) the requirement for a global lock at all.
b) geographically dispersed/unreliable network issues between two copies of
   the mailbox.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 09:57:25 +0300

> you're in for a world of hurt in a shared mailbox environment

What are you talking about? If your server implementation doesn't
reliably allow clients to access messages using sequences, it's broken.
Of course they can't permanently reference a message in different
sessions using a sequence, but within a single session the sequence <->
uid mapping can be entirely reliable all the time. And if you want to
keep it reliable when messages are removed from/added to the middle,
you'll need EXPUNGE and APPENDED replies.

> to like the single number UID because it's simple and cheap to store on

Clients also want to use UID ranges, which isn't really possible with
non-numeric and/or non-increasing UIDs. Anyway, this was already kind of
discussed with the "MSNs vs UIDs" thread I started.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Philip Guenther <[email-masked]>
Date: Fri, 22 Aug 2008 02:25:36 -0600

...
> per merge, and it just means "you need to double-check your metadata on 
> this" rather than "this is a totally brand-new message so you need to 
> fetch the lot".

Hmm, the server still has to synchronize and lock to implement 
UNCHANGEDSINCE, no?

(I suspect that if the server doesn't do that for CHANGEDSINCE too, then a 
client could catch it violating the spec by comparing concurrent 
sessions.)

Philip Guenther

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 18:15:10 +0300

Not that this has much to do with IMAP5 anymore, but..

> Hmm, the server still has to synchronize and lock to implement 

Yes, STORE UNCHANGEDSINCE requires locking to implement it correctly.
But currently there are few (if any) clients that issue it and I doubt
it's ever going to be all that widely used, so it's not that much of a
performance issue.

> client could catch it violating the spec by comparing concurrent 

I thought about this before, but I don't think it requires locking. The
server needs to synchronize the mailbox state before starting the FETCH
so it sees all the changes from the previously finished commands, but it
doesn't matter if STORE commands are allowed to run at the same time so
no locking is needed.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5