Subject: [imapext] draft-ietf-appsawg-multimailbox-search

From: Barry Leiba <[email-masked]>
Date: Thu, 6 Mar 2014 10:42:15 +0000

IMAP denizens...

Chris Newman has asked that we consider moving the Multimailbox Search
extension to Standards Track (it was published as RFC 6237 as
Experimental).  Appsawg is handling that:

http://datatracker.ietf.org/doc/draft-ietf-appsawg-multimailbox-search/

I'd like to ask the IMAP crowd to do two things:

1. Review the document and post comments to [email-masked]

2. Tell us, in your review or separately here, if you (a) have
implemented it and/or (b) have interest in implementing it (and
specify client, server, or both).

Please take a few minutes to do this -- the document isn't long, and
we very much need the reviews to move this ahead.

Thanks,
Barry

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 06 Mar 2014 21:57:53 +1100

> specify client, server, or both).

We didn't implement it because it wasn't possible to track the results
over time - instead we have a command xconvmultisearch (yes, I know,
ugly name) which:

a) only works within single users, because:
b) we keep a single highestmodseq value across all mailboxes within a single
   user, so we can
c) lock the entire user from modifications while we run the search, and
   hence ensure a consistent read, and
d) return the highestmodseq of that read, so that further requests can
   ask for updates since the previous modseq - and they'll get a list of
   anything added, removed or just touched.

This also handles the folding of related (roughly thread=references, with
some trickyness) messages into a single "exemplar".

So we're strayed quite a long way from IMAP to get what we need.

Anyway - my main problem with it is that users don't just want search - they
want sort.  Running the search across multiple mailboxes means you still need
to fetch all the data for all the matches before you can get a meaningful sort
by any parameter other than mailbox name as the first sort index.  And that
can be a lot of download to the client if you searched a common term.]

I'm not signed up for apps-discuss.  Guess I'd better go do that.

Cheers,

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 06 Mar 2014 09:30:17 -0500

Hi,

> Experimental).  Appsawg is handling that:

If sufficient servers AND clients have bothered to implement 
multimailbox-search and they have proven to be interoperable then I see no 
objection to moving to standards track (i.e, the experiment was a success 
so lets move forward) - but can we see evidence of that first?

--On March 6, 2014 at 9:57:53 PM +1100 Bron Gondwana <[email-masked]> 
wrote:

> Anyway - my main problem with it is that users don't just want search -
> they want sort.  Running the search across multiple mailboxes means you
> still need to fetch all the data for all the matches before you can get a
> meaningful sort by any parameter other than mailbox name as the first
> sort index.  And that can be a lot of download to the client if you
> searched a common term.]

On Bron's point - yes what is really needed is an extension that does away 
with the single selected mailbox model and allows searching, sorting, 
sync'ing, message access across all mailboxes all the time.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 6 Mar 2014 15:36:33 +0100

On Thursday, March 6, 2014 3:30:17 PM CEST, Cyrus Daboo wrote:
> multimailbox-search and they have proven to be interoperable 
> then I see no objection to moving to standards track (i.e, the 
> experiment was a success so lets move forward) - but can we see 
> evidence of that first?

Uh. What are some clients that have done this?

AFAICT, gmail's way (an "all mail" mailbox) has won in practice, despite 
its problems.

Arnt

---

From: Chris Newman <[email-masked]>
Date: Thu, 06 Mar 2014 16:29:46 +0000

--On March 6, 2014 9:30:17 -0500 Cyrus Daboo <[email-masked]> wrote:
> multimailbox-search and they have proven to be interoperable then I see
> no objection to moving to standards track (i.e, the experiment was a
> success so lets move forward) - but can we see evidence of that first?

I have written a server implementation of MULTISEARCH that is not presently 
released. As a server vendor, I have had two independent requests for this 
feature. At least one of those requests is from a customer that is working 
on a client implementation. I can not name the customer.

wrote:
> On Bron's point - yes what is really needed is an extension that does
> away with the single selected mailbox model and allows searching,
> sorting, sync'ing, message access across all mailboxes all the time.

I have no interest in implementing such an extension at this time. I'm 
going to assert there is not enough interest in such a proposal to get a 
specification written, thus making it not relevant to this discussion. If 
someone were to prove my assertion wrong, then we could have an interesting 
discussion about whether one or both proposals should go to standards track.

For just searching, the current MULTISEARCH design has significant benefits 
for both client and server. It produces concise results (a UID set for each 
mailbox rather than list of UIDs), and the straightforward implementation 
will return results as they are found on a per-mailbox basis in a way that 
clients can display the results as they are found. The straightforward 
implementation of MULTISORT/ALLMSGS has to compute all the results before 
sending anything to the client. There's value to having these be separate 
extensions even if sufficient interest to write a MULTISORT/ALLMSGS 
extension materializes in the future.

		- Chris

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 06 Mar 2014 11:43:03 -0500

Hi Chris,

--On March 6, 2014 at 4:29:46 PM +0000 Chris Newman 
<[email-masked]> wrote:

> I have written a server implementation of MULTISEARCH that is not
> presently released. As a server vendor, I have had two independent
> requests for this feature. At least one of those requests is from a
> customer that is working on a client implementation. I can not name the
> customer.

And the current status of experimental matters to them in terms of 
developing an implementation right now?

My point here is that until we have interoperable implementations on both 
client and server and have caught and fixed any issues in the spec, we 
shouldn't advance the status.

Well, Ok that is my interpretation of what "experimental" should mean - 
others may disagree with that...

Now I am happy to take your word for it that your customer has done a 
client implementation and is happy with the spec (or whatever changes are 
being proposed to advance it). It would be nice to hear from some other 
servers/clients as well...

FYI If I were still working on my client, I would be keen to see 
MULTISEARCH supported since it did the equivalent using select/search 
across multiple mailboxes to get results. It would definitely benefitted 
from MULTISEARCH.

-- 
Cyrus Daboo

---

From: Barry Leiba <[email-masked]>
Date: Thu, 6 Mar 2014 16:51:33 +0000

> shouldn't advance the status.

Why not?  Interoperable implementations aren't required for Proposed
Standard.  Internet Standard, sure, but not PS.  The requirement for
PS is that we think we got it right.

I would like to see more implementation experience -- it's be great to
see.  That's why I asked whether there is any.  But to me, the gating
factor is more about whether there's *interest* in implementing it.

> FYI If I were still working on my client, I would be keen to see MULTISEARCH
> supported since it did the equivalent using select/search across multiple
> mailboxes to get results. It would definitely benefitted from MULTISEARCH.

Exactly.

b

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 6 Mar 2014 19:12:19 +0200

On 6.3.2014, at 16.36, Arnt Gulbrandsen <[email-masked]> wrote:

> AFAICT, gmail's way (an "all mail" mailbox) has won in practice, despite its problems.

This is how it's been implemented in Dovecot. There's a virtual "all mails" mailbox which the clients can use with (almost) zero modifications to their code. Usually this is configured to be a hidden mailbox used only by webmails. There are also special FETCH X-MAILBOX and X-REAL-UID fields that can be used to access the underlying actual mail and SEARCH has X-MAILBOX parameter. Whenever someone has requested MULTISEARCH extension for Dovecot and I've pointed out the way to do it with virtual mailboxes, they haven't cared about MULTISEARCH anymore. So I'm not really interested in implementing it. Adding some standardization for the \All mailbox would be useful though.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 06 Mar 2014 12:12:07 -0500

Hi Barry,

> factor is more about whether there's *interest* in implementing it.

I am pretty sure people would have raised there hands saying they are 
"interested" in it before the current spec was even written. But it was 
given "experimental" status for a reason back then. Why was that? I hunted 
down the relevant message to the MORG mailing list and saw this 
(<http://www.ietf.org/mail-archive/web/morg/current/msg00507.html>):

> 2. We have changed the target status of this from standards track to
> experimental.  We think there isn't strong enough consensus to go for
> standards track, given the possible interactions with other
> extensions.

At the very least we need to address that point before we should change the 
status. And I do think that means actually having people implement it and 
see what those possible problems might be, and then report back.

-- 
Cyrus Daboo

---

From: Barry Leiba <[email-masked]>
Date: Thu, 6 Mar 2014 09:16:00 -0800

> given "experimental" status for a reason back then. Why was that?

Because we weren't sure we got it right.

> see what those possible problems might be, and then report back.

Chris, please post what you said to me when you asked me to start this
up -- the bit about how we got the interactions right.

Barry

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 06 Mar 2014 12:16:34 -0500

Hi Barry,

--On March 6, 2014 at 12:12:07 PM -0500 Cyrus Daboo <[email-masked]> 
wrote:

> At the very least we need to address that point before we should change
> the status. And I do think that means actually having people implement it
> and see what those possible problems might be, and then report back.

I note the new spec does state:

>    This extension was previously published as experimental.  There is
>    now implementation experience, giving confidence in the protocol, so
>    this document puts the extension on the Standards Track, with some
>    minor updates that were informed by the implementation experience.

I guess I am just asking for confirmation of that.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 6 Mar 2014 18:45:05 +0100

On Thursday, March 6, 2014 6:12:19 PM CEST, Timo Sirainen wrote:
> So I'm not really 
> interested in implementing it. Adding some standardization for 
> the \All mailbox would be useful though.

Such as? We do have special-use to declare \all, what more do you want?

I'd like to have something like X-GM-MSGID (and X-GM-THRID is great too).

Arnt

---

From: Chris Newman <[email-masked]>
Date: Thu, 06 Mar 2014 19:33:31 +0000

> developing an implementation right now?

The organization that I believe is implementing a client was uncomfortable 
with the experimental status. I was not given a specific reason. I suspect 
they interpreted "experimental" as meaning "subject to incompatible 
change". While that shouldn't deter implementation (and I do not believe it 
will), that would understandably make someone doing a large deployment 
uncomfortable. That's a legitimate concern and I believe moving it to 
standards track is the right response unless there's evidence the 
specification is immature. After implementing it, I found the protocol well 
designed and complete.

> shouldn't advance the status.

That's not required by the process. For this particular proposal, the 
server side is quite complex to implement so waiting for a server 
implementation before going standards track made sense. But the client side 
is just protocol from RFC 4731 (that is deployed in clients already) with 
some minor tweaks.

> servers/clients as well...

I agree it would be nice to hear from more implementers.

The two technical issues I raised were minor. I'd like the rules for 
BODY/TEXT in the context of multisearch to be loosened so that 
implementations are not required to do substring search, but could be 
considered compliant if they did a word-based indexed search (actually, I 
wouldn't mind making that change to the IMAP base spac, but that's a 
separate problem). Secondly, I'd like to simply disallow client commands 
that use the same mailbox name in two different source options. To handle 
the source options, it's necessary to build a hash table of mailbox names 
for duplicate elimination. But a source option like "IN (MAILBOXES foo 
subtree foo)" is annoying because the second "foo" can't be subject to 
duplicate elimination until after the subtree is expanded. So that means 
extra server code is needed to handle a special-case that no client ever 
needs to issue. So it's a minor tweak to simplify server implementations.

All the other issues I had were implementation issues, not protocol issues.

> across multiple mailboxes to get results. It would definitely benefitted

Thank you for saying that. I do think evidence of community interest in a 
protocol is a criteria that should be considered for moving to standards 
track. This statement counts to me as additional evidence.

		- Chris

---

From: Chris Newman <[email-masked]>
Date: Thu, 06 Mar 2014 19:59:14 +0000

> I guess I am just asking for confirmation of that.

Fair enough.

The problematic protocol interactions are with UPDATE and reuse search 
result (SAVE). After implementing I can verify that the document's approach 
to that was correct -- it limited those to the selected mailbox only. I 
encountered several complex issues implementing multisearch but they were 
not protocol issues. As far as I know we implement all the ESEARCH return 
options and I didn't have problems with the others.

I can also confirm that the source options model worked. I like it much 
better than the LIST wildcard model. The duplicate elimination was a bit 
annoying, but less annoying than multiple wildcards would be. I actually 
suspect most client use will be IN (personal) or IN (personal subscribed) 
and the latter does require duplicate elimination.

There were a lot of implementation subtleties, but they were not related to 
the protocol. Perhaps it would be helpful to mention the interaction 
between IMAP "INBOX" being case-insensitive and the duplicate elimination 
requirement, as that's an implementation detail that's impacted by the base 
spec's interaction with this.

I don't think the inboxes source option is useful. If a server implements 
sieve fileinto and variables, then inboxes is a synonym for personal. But 
synonyms are easy to implement and keeping it consistent with RFC 5465 is 
good.

I mentioned my suggested changes in a previous email.

		- Chris

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 06 Mar 2014 20:53:54 +0000

------ Original Message ------
From: "Cyrus Daboo" <[email-masked]>
>sorting, sync'ing, message access across all mailboxes all the time.
+100

actually I did some work on this approach about 9 years ago.  Just an 
extension (which I called "UNSELECTED"), which once you enter that mode, 
all msgids and UIDs in requests and responses become fully qualified - 
e.g. include the mailbox name.

I think in practise it would actually be very simple for servers to do, 
and allow notifications of changes to any message anywhere.  This would 
solve numerous problems relating to multiple clients accessing the same 
account with many folders filtering mail.  It would also make 
server-based filtering a lot more workable.

Currently I now only run 1 copy of mail that does any filtering (my work 
desktop), and remote desktop to that computer to do mail (other than iOS 
client which I don't use much to send, and don't use for filtering).  
This was because I have over 600 folders, and if one client filtered 
something, and the other didn't see it, it was effectively deleted to 
all intents and purposes - gone and unfindable to the client that didn't 
do the filtering.

Adrien

>
>-- Cyrus Daboo
>
>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 6 Mar 2014 23:36:23 +0200

On 6.3.2014, at 19.45, Arnt Gulbrandsen <[email-masked]> wrote:

>> So I'm not really interested in implementing it. Adding some standardization for the \All mailbox would be useful though.
> 

1) The \All mailbox isn't necessarily visible. GMail makes it visible, but I think it still causes problems with some clients, so in most Dovecot installations that use it it's not visible. I'm not sure if there's any nice way to fix this though. Maybe adding it as part of the capability, such as VIRTUAL=VirtualHidden/ (to support not just \All but other virtual mailboxes).

2) After virtual mailbox is already selected, ability to SEARCH messages by their original mailbox name. Dovecot currently uses X-MAILBOX that supports LIST wildcards, but MULTISEARCH's IN parameter would work just as well.

3) Ability to FETCH the original mail's mailbox name and UID.

> I'd like to have something like X-GM-MSGID (and X-GM-THRID is great too).

Yeah, those would be nice. Also the mailbox GUID if server supports those.

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 07 Mar 2014 11:18:55 +1100

> Yeah, those would be nice. Also the mailbox GUID if server supports those.

At the moment we call these DIGEST.SHA1 (the implementation is in the name,
kind of ugly.  It used to be called GUID, and it still is internally),
CID (conversation ID) and we expose the mailbox GUID as a metadata item called
/shared/vendor/cmu/cyrus-imapd/uniqueid (that bit has been upstream for ages)

All these concepts are useful - particularly X-GM-MSGID, which lets a client
discover that a different client has moved thousands of messages between folders
without re-downloading all their contents or making guesses based on message size
and the unfortunately unreliable-in-the-real-world 'Message-Id' header.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 07 Mar 2014 11:37:49 +1100

> discussion about whether one or both proposals should go to standards track.

You're asserting this based on your customer who you can't name - as opposed
to my customers who I (well, I can't really name the actual customers)... bu
the ability to sort by date overall rather than with each mailbox was a commonly
requested search feature at FastMail - and we now support it.

We definitely don't have anyone asking us to go back to sorting by mailbox
only - particularly since mailbox name is still a sort key that you can use
in your sort if you want to.

> extension materializes in the future.

Our result set is still quite concise - though it's tuples:

3 xconvmultisort (reverse arrival) (conversations position (1 30)) utf-8 FUZZY TEXT "chris.newman" not folder "Inbox.Trash" not folder "Inbox.Junk Mail"
* XCONVMULTI (("INBOX" 1148523981) ("INBOX.Archive.Inbox-2011" 1298369152) ("INBOX.Drafts" 1148523983) ("INBOX.Lists.Cyrus" 1193627117) ("INBOX.Lists.Imap5" 1217730348) ("INBOX.Lists.ImapExt" 1198790887) ("INBOX.Lists.fm-cvs" 1148523992) ("INBOX.Sent Items" 1148523995)) ((75e35b068901c51b (2 20195) (5 5210) (5 5209) (5 5207) (5 5206) (5 5205) (5 5203) (5 5202) (5 5201) (5 5199) (5 5198) (7 13690) (5 5197)) (22e8c6bc0c999471 (5 5106) (0 946541) (5 5104) (7 13234) (5 5102)) (35f42610ba6324f3 (5 5079)) (42599c237f434f9f (5 5078) (5 5077) (5 5076) (5 5074) (5 5073) (5 5072) (5 5071)) (94643209768ba85c (5 5070) (5 5069) (5 5068)) (a85bf44cea5ba267 (6 33423)) (8980e46f8cc9617e (6 33248)) (7bec218f48a35592 (6 30446)) (f6375654cf52cb5e (6 28692)) (0cc65f89dc003eb7 (7 9754)) (df7c52435658163e (1 5246) (1 4990) (1 4987) (1 4967)) (14744ad604794672 (1 3190) (4 201) (7 8057) (1 3183)) (77cb8893eeb61bb4 (3 13491)) (c613c22cd60fc734 (4 54) (4 53) (4 51) (4 47)) (cd0f24d8cd2da0e3 (4 50) (4 49) (4 48) (4 46)) (8c1c6d50a25d7d75 (4 41)) (aa0a6113c53456ea (5 4315)) (da7343dea9a2dd92 (5 4308) (5 4303)) (7f0aa97dba576376 (5 4286)) (86631b9c7fd03ba8 (5 4269)) (1dc2b83458b9edb4 (5 4267)) (764d53b27438df5d (5 4261)) (4657718a320253b3 (5 1023)) (8216063d1d06d1a2 (5 3801) (5 1857) (5 1649) (5 3605) (5 987) (5 1453) (5 2782) (5 3152) (5 622) (5 707) (5 2587) (5 2670) (5 427) (5 2393) (5 3273) (5 1109) (5 511) (5 1395) (5 3359) (5 1201) (5 3065) (5 897)) (885c99dc1324a315 (5 3977) (5 1218) (5 3172) (5 1006) (5 2960)) (86e821cd553f0665 (3 2228)) (40dec021413028d4 (5 4046)) (d2a6ca5cfc0cd33a (5 4098) (5 1939) (5 1280) (5 2153) (5 3885) (5 1726) (5 3542) (5 1389)) (5e0e810635ad3ebf (5 793) (5 1666) (5 3624) (5 1470)) (51cbbb00a51db66d (5 1383) (5 3348) (5 2384) (5 225) (5 1102) (5 3057) (5 672) (5 1558)))
* OK [POSITION 1]
* OK [HIGHESTMODSEQ 295339132831582]
* OK [TOTAL 109]
3 OK Completed (in 0.110 secs)

Which breaks down to:

a list of folder names and uidvalidities.

a list of the first 30 matching conversations, each of which is (CID (tuple) (tuple) ...)

each tuple is folder index (from the list above) and UID within that folder.  This is used to hilight the matching messages within the conversation when it is opened.

... and if you scroll past that first 30 item window, you can fetch more.  Obviously, there's no need to work in small windows - it's just an option that we use in our API to avoid getting unbounded amounts of data returned to the client.

4 xconvmultisort (reverse arrival) (conversations multianchor (1383 Inbox.Lists.ImapExt 1 30)) utf-8 FUZZY TEXT "chris.newman" not folder "Inbox.Trash" not folder "Inbox.Junk Mail"
* XCONVMULTI (("INBOX" 1148523981) ("INBOX.Archive.Inbox-2011" 1298369152) ("INBOX.Drafts" 1148523983) ("INBOX.Lists.Cyrus" 1193627117) ("INBOX.Lists.Imap5" 1217730348) ("INBOX.Lists.ImapExt" 1198790887) ("INBOX.Lists.fm-cvs" 1148523992) ("INBOX.Sent Items" 1148523995)) ((a7c22d265a01ae47 (5 1808) (5 2682)) (0e7f42859d14064b (5 522)) (18683122949a755f (5 2482)) (839b47557488a2f0 (5 2008) (5 2282) (5 111)) (17d7c11d5c313214 (5 3578) (5 650)) (ddf8dad81171c58f (5 4005)) (dd20935a4d7b2534 (5 1476)) (b4bbc805e5d88997 (5 811)) (d4b09f52f1551e51 (5 1955)) (9d1382e572868684 (5 4025) (5 1865)) (0285ab40d0125d6e (5 3534)) (b84d7c33ee0ce08b (5 2772) (5 612)) (2faf629f598823c9 (5 2873)) (16c0037968ce0d5d (5 1393) (5 3357)) (320995b4ec6d6c99 (5 1741)) (52f526d95c7a85f5 (5 1459) (5 3419)) (a45377efe133440c (5 3500)) (682cd6d68615f357 (5 1997)) (cd77fa3872d98167 (5 501) (5 2661) (5 2463) (5 302)) (f6f2cd14f1e19e6f (5 2926) (5 1190) (5 3141) (5 975) (5 753) (5 2121)) (f7ff660331ebbe9d (5 3053)) (d01b067ac95c0f9d (5 1696)) (24ce7fe1b6913094 (5 1621) (5 3577)) (5b5bbd4ce7ce019a (5 3497)) (472840f7906f56a2 (5 1136) (5 1797)) (aa51277478e5d2a2 (5 3002) (5 3866)) (dfcd789dbf52d20d (5 3745) (5 2070) (5 2728) (5 570) (5 2530)) (780c5432ead295aa (5 1710) (5 3665) (5 1512) (5 3469) (5 239)) (d9a39a69640a0151 (5 873)) (d70e71aab74630dd (5 2821)))
* OK [POSITION 31]
* OK [HIGHESTMODSEQ 295339132831585]
* OK [TOTAL 109]
4 OK Completed (in 0.070 secs)

... this is a search run on our production system against my folders, using the xapian FUZZY indexed full-message search over what expands to about 6 different backend indexes spread over slow disk (compressed long-term index), slow disk (weekly compressed index), ssd (daily index updates) and finally tmpfs (real-time indexing of new messages).

So this is working now... it's insanely non-standard, but it's a much nicer user experience than the alternatives - including multisearch.

It gives me a list of conversations ordered by most recent message.

If you weren't doing conversations, the syntax would simplify to just a list of folder names that matched (with uidvalidity) and then a list of tuples of (index uid) mapping into that folder listing.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 7 Mar 2014 09:21:31 +0100

On Thursday, March 6, 2014 10:36:23 PM CEST, Timo Sirainen wrote:
> 1) The \All mailbox isn't necessarily visible. GMail makes it 
> visible, but I think it still causes problems with some clients,

Clients that issue fetch 1:* for all mailboxes.

> I'm not sure if there's any nice way to fix this though.

1. I think there isn't. I know two non-nice ways. I think I'd rather just 
be blunt about it in the document. "If you try to download all the mail, 
you should be aware that about 5% have more than 2GB and 0.01% of users 
have more than 20GB. The percentages are likely higher among those who use 
this extension." (Those numbers are just informed guesses, though.)

2. This is an intrinsic problem, not a feature of \all. If you allow 
clients a way to fetch everything, any way, ones that do fetch everything 
will have problems.

> Yeah, those would be nice. Also the mailbox GUID if server supports 
those.

Indeed.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Fri, 07 Mar 2014 13:23:24 -0800 (PST)

> AFAICT, gmail's way (an "all mail" mailbox) has won in practice, despite

I have to say I find this entire conversation to be confusing in the extreme.

We have an experimental RFC, implementation evidence that it qualifies for
standards track status, and a desire for it to have that status.

And the argument against is that there are at least two other ways to do it,
neither of which, AFAICT, is specified anywhere. And if someone has volunteered
to write up either alternative and see it through the standards process, I
missed seeing that offer. (*)

So we're left with something clients have a real need to do, an experimental
specification that provides one way to do it, a willingness to move it to
standards track, and a bunch of ad-hoc deployment. And the response seems to be
to oppose doing anything?

				Ned

(*) And before anyone says that the \All business is "obvious", there have
already been messages in this thread pointing out that it really isn't. And
even if it requires no protocol changes whatsoever, the model needs to be
documented.

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 07 Mar 2014 16:54:08 -0500

Hi Ned,

> missed seeing that offer. (*)

To clarify my position - whilst I advocated for a more generic "get to all 
mailboxes" extension, I do not want that to be an argument against 
multimailbox search which, as I have said, I consider beneficial. And 
frankly, I do not believe we will ever get the more generic extension 
anyway, and certainly don't have the time or inclination to work on it.

At this point I am satisfied with Chris' technical commentary on 
implementation of multimailbox search. I would still like to hear from 
others, but I have no objection to upping the status.

-- 
Cyrus Daboo

---

From: Ned Freed <[email-masked]>
Date: Fri, 07 Mar 2014 14:16:36 -0800 (PST)

> others, but I have no objection to upping the status.

OK, this makes perfect sense to me. Apologies for any misinterprtation I
made of your position in my original message.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 8 Mar 2014 09:16:11 +0100

Personally speaking, I'm not opposed to advancing multimailbox search. I 
think it's a little bit pointless since dovecot and gmail dominate the 
landscape and both already support \all. But if you want to do it, go 
ahead.

Arnt

---

From: Chris Newman <[email-masked]>
Date: Mon, 10 Mar 2014 20:43:11 +0000

It sounds like IMAP cross-folder sort is a big non-interoperable mess right 
now due to lack of standardization.

Have any of you considered writing a specification of your multisort 
protocol? Or maybe arrange a phone call between proponents of the 
"xconvmultisort" proposal, the "X-MAILBOX, X-REAL-UID" and the "\all" 
proposals to see if consensus on a standard approach is possible?

Standards have several benefits for the vendors who participate:

1. If you have an early implementation, that's a key marketing/selling 
point.

2. It gives your customers more choice of the other side when they choose 
your product. (more client options if you're a server vendor, or vice-versa 
if you're a client vendor).

3. It improves your reputation. It shows you care about the future of your 
customers and the future of the Internet.

4. It raises the innovation baseline. Once a standard is written and 
successful, it becomes "common practice" over time. Then there's no more 
need to proselytize that standard, new innovation can start with that 
standard as a baseline.

5. Once a standard is approved, you're much less likely to have to waste 
time and effort implementing two or more ways of doing the same thing to 
interoperate with different implementations.

6. It's one way a smaller vendor can get a leg-up on a bigger vendor who 
ignores standards.

While private IMAP extensions are fine for experiments, they're an 
interoperability problem when more than one for the same purpose show up. 
Why not take the time to write and progress a protocol specification so you 
and your customers reap the benefits and the community's innovation 
baseline is raised?

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 19 Mar 2014 13:50:54 +1100

On Tue, Mar 11, 2014, at 07:43 AM, Chris Newman wrote:
> It sounds like IMAP cross-folder sort is a big non-interoperable mess
> right now due to lack of standardization.
>

Apologies for the late reply on this - I've been busy dealing with a
nasty server partial-failure which led to corruption, and the associated
cleanup and recovery.  The "joy" of devops.

> 2. It gives your customers more choice of the other side when they
>    choose your product. (more client options if you're a server
>    vendor, or vice-versa if you're a client vendor).
>
> 3. It improves your reputation. It shows you care about the future of
>    your customers and the future of the Internet.
>
>    successful, it becomes "common practice" over time. Then there's no
>    more need to proselytize that standard, new innovation can start
>    with that standard as a baseline.
>
> 5. Once a standard is approved, you're much less likely to have to
>    waste time and effort implementing two or more ways of doing the
>    same thing to interoperate with different implementations.
>
> 6. It's one way a smaller vendor can get a leg-up on a bigger vendor
>    who ignores standards.
>
> interoperability problem when more than one for the same purpose show
> up. Why not take the time to write and progress a protocol
> specification so you and your customers reap the benefits and the
> community's innovation baseline is raised?

We have indeed.  Greg Banks (who doesn't work at FastMail any more, but
wrote the initial conversations support) did some initial work towards a
standards proposal for our conversations.  It was paused mainly because
we wanted to iron out the issues which would become obvious with real-
world usage first.

We have also done some work towards a more general mail access protocol
over HTTP, which is shared at http://jmap.io/

My general guiding principle is that the client should be describing
what it wants as clearly as possible, and the language of the protocol
needs to have the facility for it to do so.

The secondary principle is that the server should never do work that the
client has not explicitly requested.

This means that MSGNO by default is out, because that's an MVCC snapshot
of mailbox data.  It's an ordering that existed at a point in time, and
needs to be maintained by the server for an unbounded length of time.
The client can issue commands every 29 minutes which are chosen to not
allow EXPUNGE notifications - and the server can't forget those messages
until the mailbox is unselected.

Also, UNSEEN and RECENT have costs to calculate which aren't required by
many clients.

One thing that really frustrates me is optimisation hacks encoded in the
standard and guidance.  Things like "do a listing with %, then recurse
into each sub mailbox".  It might be more efficient when you're talking
to a server which is storing the folders in a filesystem and doing the
moral equivalent of 'ls' on each directory.

It's significantly less efficient with Cyrus, where the mailboxes database
is quite efficient to iterate in mailbox order (including sub-folders).
You're basically inspecting the sub-folders anyway - at least until you
can generate the \HasChildren or otherwise.

Generally, I would like the client to be able to say "I want the full
folder tree thanks, but limit it to the first 1000 items".

So my ideal protocol for a X-MULTISEARCH would smell a bit more like SQL
I suspect.  The offset / limit or anchor / limit syntaxes are both very
useful ways of describing exactly what you want to the server.

------

It would also be really great to merge the X-GM-MSGID and similar tricks
that various sites are using to uniquely identify messages across copy/move
operations that were performed by a DIFFERENT client.

APPENDUID is all very nice if you're using the client which did the COPY,
but it falls down if a different client did the copy - you wind up having
to fetch the entire message again if you don't trust the BODYSTRUCTURE and
ENVELOPE to uniquely identify the message enough - and most clients don't
due to the fact that Message-Id isn't 100% reliable in this broken age :(

I think those are the two areas.  I've said before that I'm willing to put
the time in to do this - and you're right - I really should stop talking
and start producing!

------

One final thing that we have found insanely useful in Cyrus replication:
checksums all over the place.  This is not so much useful to a traditional
"partial view" client, but many clients seem to synchronise the entire
contents of a mailbox, and a CRC would have value there.

The issue that's kept me from having time to reply for the past week is
documented here:

http://blog.fastmail.fm/2014/03/13/cleaning-up-from-an-imap-server-failure/

Having the sha1 of the spool files in our logs enabled data recovery in a
way that would have otherwise been impossible on an untrusted filesystem.
We could be sure that the emails we recovered were not corrupted.

Having a strong digest available for every message on the server would also
allow clients to consistency-check their local cache without resyncing all
the data.  I used to think that being able to detect mailbox renames that
you didn't do with this client was important (right now if another client
renames a folder, your client needs to fetch all the messages again for
the same reason as message copies) - but I think re-fetching just the flags/
annotations/digest isn't too expensive if you can re-use the cached message
contents.

Bron.

-- 
  Bron Gondwana
  [email-masked]