Subject: [MORG] I-D Action:draft-ietf-morg-multimailbox-search-03.txt

From: [email-masked]
Date: Fri, 18 Jun 2010 13:15:02 -0700 (PDT)

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Message Organization Working Group of the IETF.

	Title           : IMAP4 Multimailbox SEARCH Extension
	Author(s)       : B. Leiba, A. Melnikov
	Filename        : draft-ietf-morg-multimailbox-search-03.txt
	Pages           : 10
	Date            : 2010-06-18

The IMAP4 specification allows the searching only of the selected
mailbox.  A user often wants to search multiple mailboxes, and a
client that wishes to support this must issue a series of SELECT and
SEARCH commands, waiting for each to complete before moving on to the
next.  This extension allows a client to search multiple mailboxes
with one command, limiting the round-trips and waiting for various
searches to complete, and not requiring disruption of the currently
selected mailbox.  This also uses MAILBOX and TAG fields in ESEARCH
responses, allowing a client to pipeline the searches if it chooses.

Note

A revised version of this draft document will be submitted to the RFC
editor as a Proposed Standard for the Internet Community.  Discussion
and suggestions for improvement are requested, and should be sent to
[email-masked].

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-morg-multimailbox-search-03.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Barry Leiba <[email-masked]>
Date: Fri, 18 Jun 2010 16:40:32 -0400

> http://www.ietf.org/internet-drafts/draft-ietf-morg-multimailbox-search-03.txt

This document is almost done, but still has a few open issues, noted
in the "open issues" section.  Timo responded in February:
   http://www.ietf.org/mail-archive/web/morg/current/msg00308.html
...but there was no discussion of his comments.  I want to finish this
and resolve the open issues, so everyone, please help with this.

>> Interaction with IMAP Contexts [RFC5267] needs to be defined.
>
> I guess server then just has to monitor all matched mailboxes and send
> updates whenever they happen. But should it notice newly created/renamed
> mailboxes? Maybe the answer is no, except if NOTIFY is also used:
>
>>    Suggestion: UPDATE option of IMAP Notify [RFC5465] might have to be
>>    prohibited when both Context and this extension are used.  [Barry:
>>    But why?  Isn't this then the equivalent of doing many SEARCH
>>    commands with the UPDATE option?]
>
> I guess the problem is if NOTIFY's extended UPDATE should be allowed.
> Sending FETCH replies for messages matching in non-selected mailboxes
> wouldn't work. But the solution could be just that the FETCH replies are
> sent only to messages in selected mailbox, whichever it happens to be at
> the time changes happen.

Does anyone have other comments here?  Can anyone suggest text, if
there's text that needs to be added?

>> There's a suggestion to add an EXCLUDE clause (with a nested search
>>    criteria).  For example, one might want to search "personal" but
>>    exclude "mailboxes (Trash Spam)".
>>
>>    Cyrus suggests you can use metadata to tag mailboxes that you do or
>>    don't want to search, and use search criteria to exclude them (good
>>    for mailboxes that you NEVER want searched (as Trash and Spam,
>>    above)).
>
> Dunno. A 3rd possibility would be to make the IN a search-key, but
> perhaps that would make things too complex again. It would anyway allow
> e.g.:
>
> SEARCH unseen IN (personal) NOT IN (mailboxes "Spam" subtree "Trash")

My sense here is that we can (and should) defer this to a future
extension, if it turns out to be needed.  My sense is that it isn't
strongly needed.

>> C: tag1 SEARCH IN ((mailboxes "folder1" subtree "folder2")) unseen
>>    C: tag2 SEARCH IN ((mailboxes "folder1" subtree-one "folder2"))
>
> Double (( )) doesn't seem correct?

Can someone check the formal grammar for me?  I wrote the text based
on what I saw in the grammar, and my eyes are now glazing over on it.
Help, please, to make sure it's right.

Barry, as document editor

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 18:01:02 -0700 (PDT)

> >        Filename : draft-ietf-morg-multimailbox-search-03.txt

Looking back at the discussion of the previous draft, I'm with Arnt.
I'm really not comfortable with allowing MSNs in the UID SEARCH
criteria of a multimailbox search.  UIDs are designed as persistent
outside the context of a SELECT/EXPUNGE.  MSNs are expressly tied
to a SELECT/EXPUNGE session.  Absent a session, there's no sequence
to reference.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Jun 2010 18:22:15 -0700 (PDT)

> to a SELECT/EXPUNGE session.  Absent a session, there's no sequence

I'm quite sympathetic to this argument, but...

Actually, there is a (limited) use for MSNs in the UID SEARCH criteria.

An MSN range of 1:100 explicitly limits the results to the first 100
messages.  Note that unlike sequences in FETCH etc. commands, SEARCH
criteria are not syntactically required to respect the EXISTS count.  I
also note that this was the justification for MSNs to be in single-mailbox
SEARCH and UID SEARCH.

I also worry that removing MSNs creates multiple syntax flavors of SEARCH
criteria.  Implementations may have to go to trouble to remove MSN
capability from multi-mailbox search since MSN capability is mandatory to
implement in single mailbox search.

I recommend, instead, that there be a note to the effect that MSNs are not
particularly useful in multi-mailbox search; and that the use for MSNs in
multi-mailbox search, such as it is, is probably in sequence ranges for
limiting.

To summarize, then:

[1] I agree that MSNs in SEARCH criteria are of very limited use.

[2] However, that use is nearly as applicable to multi-mailbox search as
it is to single mailbox search.

[3] Leaving MSNs in multi-mailbox search probably doesn't do any harm, as
the engine already has to support them for single mailbox search.  At
worst, they are useless and harmless.  At best, they are harmless and
largely useless except for a very obscure case.

[4] Removing MSNs requires a new syntax rule that excludes MSN and
consequent greater complexity.  That's a harm.

In my judgment, the harm from [4] outweighs the uselessness of [3].  If
this was a proposal to add MSN support, as opposed to deleting it, I would
take the opposite side.

Thus, leave it in with a note about their limited use case and being in
there to correspond with single-mailbox search.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 19:50:54 -0700 (PDT)

> Actually, there is a (limited) use for MSNs in the UID SEARCH

I've got no problem with MSNs in UID SEARCH queries on the selected
mailbox.  Since the mailbox is selected, there is a sequence.  Since
there's a sequence, you can constrain a search with it.

In the multimailbox search, however, there is no sequence for the
non-selected folders.

> I also worry that removing MSNs creates multiple syntax flavors of
> SEARCH criteria. Implementations may have to go to trouble to remove
> MSN capability from multi-mailbox search since MSN capability is
> mandatory to implement in single mailbox search.

But it's not a given that an IMAP implementation will use the exact
same code path to do multimailbox SEARCH as to do vanilla SEARCH.
And MSNs are the point where the code would likely diverge.

Would it be reasonable to define a command other than UID SEARCH to
represent multimailbox search?  The IN clause would then be defined
on the new command and not on UID SEARCH (probably a plus), and we
could constrain the set of allowable search criteria without
introducing incompatibilities with base SEARCH.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Jun 2010 20:11:23 -0700 (PDT)

> In the multimailbox search, however, there is no sequence for the

1:100 is a meaningful concept even in multi-mailbox search.  It means
"only consider the first 100 messages in any mailbox".

> And MSNs are the point where the code would likely diverge.

Why?

> introducing incompatibilities with base SEARCH.

How much do you want it generally implemented?

Syntax complexity and special rules are not your friend.

-- Mark --

http://panda.com/mrc

---

From: Barry Leiba <[email-masked]>
Date: Tue, 22 Jun 2010 23:24:17 -0400

> In the multimailbox search, however, there is no sequence for the

Well, there is -- it's just that the client doesn't know what messages
correspond to what MSNs, directly.  That makes the utility low, but
not absent.

I think Mark's example is a little weak -- a client would be more
likely to want to limit a search to the 100 newest messages, rather
than the 100 oldest, and "*-100" doesn't work -- but I agree with him
that it's worth some words in the document (to the effect that clients
are likely to find little use for MSNs here) rather than an outright
restriction.

> And MSNs are the point where the code would likely diverge.

That seems a stretch.  I can't imagine implementing this search as
anything other than multiple applications of a regular search, perhaps
spinning off a few threads to do some in parallel.

> represent multimailbox search?

We did talk about that a long time ago ("MSEARCH", or some such) and
ruled it out.  It's much more likely to be implemented this way, and
it's inviting trouble to have too many similar but not-quite-the-same
commands.

I'm happy to go either way on this if we have consensus to change it,
but my preference would be to leave MSNs in, and to add text
explaining why clients should generally avoid using them with this
extension.

Barry

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 20:25:25 -0700 (PDT)

> "only consider the first 100 messages in any mailbox".

I'd respectfully disagree.  The sequence is an ordering of the messages
in the currently-selected mailbox.  A sequence for one SELECT may not
match the sequence for a concurrent SELECT from another client due to
expunges.  And, for that reason, "1:100" does not mean the first 100
messages in the selected mailbox; rather, it means the first 100
entries in the sequence.

> Syntax complexity and special rules are not your friend.

I think that you're overstating the difficulty here.  If a server
implementor is going to do the heavy lifting required to implement 
multimailbox search, minor differences in parsing rules (like the IN
clause) are going to be the least of their worries.

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 20:31:22 -0700 (PDT)

> spinning off a few threads to do some in parallel.

Actually, I can think of a few tricks particular to my server's
implementation that might allow me to write multimailbox search without
loading each target mailbox in turn and running a vanilla UID SEARCH.
I haven't tried coding it up yet, but I *think* I see how it could be
done.

In other words, your mileage may vary.  But it's not a given that
everyone will implement multimailbox search identically.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Jun 2010 20:36:18 -0700 (PDT)

> messages in the selected mailbox; rather, it means the first 100

That is not what a sequence means in search.

 	tag SEARCH 1:100
is valid even if there are only 14 messages in the mailbox.

If your implementation does not do this correctly, it is broken.

> clause) are going to be the least of their worries.

Hardly anyone implements more than a tiny percentage of the available IMAP
extensions out there.

There are barriers to implementation, and each incremental barrier makes
it worse.

-- Mark --

http://panda.com/mrc

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 20:38:29 -0700 (PDT)

> > I'd respectfully disagree. The sequence is an ordering of the
> > messages in the currently-selected mailbox. A sequence for one
> > SELECT may not match the sequence for a concurrent SELECT from
> > another client due to expunges. And, for that reason, "1:100"
> > does not mean the first 100 messages in the selected mailbox;
> > rather, it means the first 100 entries in the sequence.
> 

I'm not arguing as to whether or not it is *valid*.  I am discussing

---

From: Dan Karp <[email-masked]>
Date: Tue, 22 Jun 2010 20:44:36 -0700 (PDT)

> I'm not arguing as to whether or not it is *valid*. I am discussing

Oops.  I accidentally hit the "send" button.  Apologies.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Jun 2010 20:46:09 -0700 (PDT)

On Tue, 22 Jun 2010, Dan Karp wrote:
> I'm not arguing as to whether or not it is *valid*.

It is valid and it means the first 100 messages in the mailbox.

In a multi-mailbox search, it would refers to the first 100 messages in
each mailbox being searched for the same reason that UID 123 refers to a
message with that UID in the mailbox being searched and not UID 123 in the
currently selected mailbox.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Tue, 22 Jun 2010 21:06:53 -0700 (PDT)

On Tue, 22 Jun 2010, Barry Leiba wrote:
> likely to want to limit a search to the 100 newest messages

Agreed.  But there are some real-life use cases that I have dealt with in
the past where it had use.

These are not enough to argue for inclusion; but we're not arguing
inclusion.  We're arguing exclusion; the path of complexity is to have a
separate syntax that excludes MSN in search.

> explaining why clients should generally avoid using them with this

That is my preference as well.

-- Mark --

http://panda.com/mrc

---

From: Barry Leiba <[email-masked]>
Date: Wed, 23 Jun 2010 13:14:16 -0400

> That is my preference as well.

Would anyone else like to weigh in, besides Dan, Mark, and me?

Barry

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 23 Jun 2010 18:54:58 +0100

>Would anyone else like to weigh in, besides Dan, Mark, and me?
I am on the fence and can go either way.

But I don't want to have a separate search criteria syntax for MSEARCH 
(or whatever it ends up being called).

---

From: Dan Keen <[email-masked]>
Date: Wed, 23 Jun 2010 14:09:31 -0700

> Would anyone else like to weigh in, besides Dan, Mark, and me?

I think we should leave them in.  They add arguable value in ranged searches.  They don't really hurt anybody.

They may make implementations easier, and they make the language simpler by retaining consistency.

I wouldn't personally use them (MSN ranges are the opposite direction from user interaction for us), but that's no reason to take a stance against them.

D

---

From: Barry Leiba <[email-masked]>
Date: Wed, 23 Jun 2010 17:22:43 -0400

We've had a good bit of discussion on the MSN issue, and I'm happy to
see it continue.

Unfortunately, there's been no discussion of the to-do list, or of

> Help, please, to make sure it's right.

We need comments and consensus on this stuff in order to move the
document forward.

Barry, as chair

---

From: Dan Karp <[email-masked]>
Date: Sat, 26 Jun 2010 12:27:51 -0700 (PDT)

> on what I saw in the grammar, and my eyes are now glazing over on
> it. Help, please, to make sure it's right.

It's incorrect.  It should have single parens instead.

   search-source-opts =  "IN" SP "(" source-mbox [SP
           "(" scope-options ")"] ")"

   source-mbox =  filter-mailboxes *(SP filter-mailboxes)

   filter-mailboxes = filter-mailboxes-selected /
                   filter-mailboxes-other

   filter-mailboxes-selected =  "selected"

   filter-mailboxes-other = "inboxes" / "personal" / "subscribed" /
                   ( "subtree" SP one-or-more-mailbox ) /
                   ( "mailboxes" SP one-or-more-mailbox ) /
                   ( "subtree-one" SP one-or-more-mailbox )

   one-or-more-mailbox = mailbox / many-mailboxes

   many-mailboxes  = "(" mailbox *(SP mailbox) ")"

---

From: Dan Karp <[email-masked]>
Date: Sat, 26 Jun 2010 17:27:00 -0700 (PDT)

> http://www.ietf.org/internet-drafts/draft-ietf-morg-multimailbox-search-03.txt

There should probably be a note about interaction with the
SEARCHRES extension.  Is it disallowed?  Does it just save
the search results in the currently selected mailbox?

---

From: Barry Leiba <[email-masked]>
Date: Mon, 28 Jun 2010 16:39:30 -0400

>           "(" scope-options ")"] ")"
...
>   many-mailboxes  = "(" mailbox *(SP mailbox) ")"

Thanks.  Yes, it was the "many-mailboxes" specification that confused
me. but that would result in something like this:
    C: tag2 SEARCH IN (mailboxes ("folder1" "folder2"))
...which still isn't close to what I had.  I'll fix the example.

Barry