Subject: [imapext] CONDSTORE interaction with SORT/THREAD

From: Alexey Melnikov <[email-masked]>
Date: Mon, 12 Aug 2013 12:56:17 +0100

Timo and Dan pointed out earlier that CONDSTORE/QRESYNC don't specify 
how they interact with SORT and THREAD. In particular, what kind of 
response should be used for requests like

  A001 SORT (DATE) UTF-8 MODSEQ 500

We have a couple of choices here and I would like to know what people's 
preferences are, in particular if people implement any particular choice 
already.

For SORT that includes a MODSEQ search criterion, possible options are:

1). Use ESEARCH response (RFC 4731) to return MODSEQ for all matching 
messages and also extend it to return sorted results. (This can be 
potentially be used for SEARCH with MODSEQ search criterion as well). 
For example:

  * ESEARCH (TAG "A001") SORTED 3,4,1,2 MODSEQ 175

Here SORTED is similar to ALL, but it preserves the SORT order of 
matching messages.
RFC 4731 already defines MODSEQ return item.

2). Extend SORT response in the same way CONDSTORE extended SEARCH response:

  * SORT 5 3 4 1 2 (MODSEQ 175)

3). Return 2 response: an ESEARCH with MODSEQ and a SORT:

  * SORT 5 3 4 1 2
  * ESEARCH (TAG "A001") MODSEQ 175

(they can be returned in any order)

I personally have a slight preference for choices #3/#1, as I don't want 
to prolifirate new variants of responses which are not ESEARCH based.

Similarly, for THREAD:

1). Use ESEARCH response (RFC 4731) to return MODSEQ for all matching 
messages and also extend it to return threaded results. For example:

  * ESEARCH (TAG "A002") THREAD ((2)(3 6 (4 23)(44 7 96))) MODSEQ 909240921

2). Extend THREAD response in the same way CONDSTORE extended SEARCH 
response:

  * THREAD (2)(3 6 (4 23)(44 7 96)) (MODSEQ 909240921)

3). Return 2 response: a ESEARCH with MODSEQ and a THREAD:

  * SORT 5 3 4 1 2
  * THREAD (2)(3 6 (4 23)(44 7 96))
  * ESEARCH (TAG "A002") MODSEQ 909240921

Here again, I prefer choices #1/#3.

If people implemented other choices, please let me know.

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 12 Aug 2013 16:27:47 +0300

On 12.8.2013, at 14.56, Alexey Melnikov <[email-masked]> wrote:

> Timo and Dan pointed out earlier that CONDSTORE/QRESYNC don't specify how they interact with SORT and THREAD.

I don't remember, I guess it was a long time ago.

> In particular, what kind of response should be used for requests like
> 
> We have a couple of choices here and I would like to know what people's preferences are, in particular if people implement any particular choice already.
> 
> 1). Use ESEARCH response (RFC 4731) to return MODSEQ for all matching messages and also extend it to return sorted results. (This can be potentially be used for SEARCH with MODSEQ search criterion as well). For example:
> 
> Here SORTED is similar to ALL, but it preserves the SORT order of matching messages.

This could be a nice new option, but I don't think MODSEQ parameter alone should change the reply so radically.

> * SORT 5 3 4 1 2 (MODSEQ 175)

Dovecot has been returning this for last 4 years (since v1.2), so it's already quite widely used.. I don't think I actually intended to add it, the SEARCH and SORT code was just mostly shared and it just happened. So this would be my recommendation. :)

> I personally have a slight preference for choices #3/#1, as I don't want to prolifirate new variants of responses which are not ESEARCH based.

One possibility would be to return #1, but only if some other new parameter is used. That parameter could then also be used without MODSEQ. It also wouldn't break with the existing Dovecot installations.

> Similarly, for THREAD:
> 
> 1). Use ESEARCH response (RFC 4731) to return MODSEQ for all matching messages and also extend it to return threaded results. For example:
> 

With THREAD Dovecot doesn't return MODSEQ in any way.

---

From: Jan Kundrát <[email-masked]>
Date: Mon, 12 Aug 2013 15:32:17 +0200

On Monday, 12 August 2013 13:56:17 CEST, Alexey Melnikov wrote:
> 1). Use ESEARCH response (RFC 4731) to return MODSEQ for all 
> matching messages and also extend it to return sorted results. 
> (This can be potentially be used for SEARCH with MODSEQ search 
> criterion as well). For example:
>

I've just re-read the relevant bits of RFC 4551 again. I have to admit I do 
not understand the point of including the highest MODSEQ of the matching 
messages under the MODSEQ key -- what is the use case for that? Sure, it's 
in the RFC so we cannot change it now, but I'd like to understand the 
motivation behind this.

> RFC 4731 already defines MODSEQ return item.

Unless I'm missing something, the example response invents a non-existant 
"SORTED" ESEARCH return value. RFC 5267 (CONTEXT=...) already defines the 
ESORT extension and the ESORT command and how it uses the ESEARCH response 
(some servers were broken and used to return an ESORT response, btw). The 
result is passed as if it was triggered via the ESEARCH command, only in a 
sorted form, so the correct return data are, IMHO:

* ESEARCH (TAG "A001") ALL 3,4,1,2 MODSEQ 175

I'm also confused because the command asked for MODSEQ >= 500, but the 
response claims that the highest MODSEQ of any matching messages is 175.

>   * SORT 5 3 4 1 2 (MODSEQ 175)

That would require a modification of the ABNF and as such it cannot be 
introduced retroacitvely, right?

>   * ESEARCH (TAG "A001") MODSEQ 175

I realize this is about SORT, not SEARCH, but RFC 4466, section 2.6.1 
contains a SHOULD saying that there should only be a SEARCH *or* an ESEARCH 
response, not both.

> I personally have a slight preference for choices #3/#1, as I 
> don't want to prolifirate new variants of responses which are 
> not ESEARCH based.

Indeed, and for the reasons stated above, I prefer 1) after it's fixed to 
use ALL instead of SORTED.

> 1). Use ESEARCH response (RFC 4731) to return MODSEQ for all 
> matching messages and also extend it to return threaded results. 
> For example:
>

I would love to have the THREAD return option for ESEARCH; in fact, I've 
tried to introduce it in draft-kundrat-incthread [1]. It will help Trojita 
understand the matching between commands and responses and therefore enable 
better pipelining.

With kind regards,
Jan

[1] http://tools.ietf.org/html/draft-kundrat-incthread-01#section-2.3.1

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 13 Aug 2013 17:57:49 +0100

> Dovecot has been returning this for last 4 years (since v1.2), so it's already quite widely used.. I don't think I actually intended to add it, the SEARCH and SORT code was just mostly shared and i…
While I dislike this option, I don't see a reason to break your 
implementation, as long as other common implementations didn't do 
something different...
> One possibility would be to return #1, but only if some other new parameter is used. That parameter could then also be used without MODSEQ. It also wouldn't break with the existing Dovecot installa…
Right. For example RESMODSEQ (it has the same semantics as MODSEQ 
SEARCH/SORT item, but causes ESEARCH response to be returned).

> With THREAD Dovecot doesn't return MODSEQ in any way.
Ok. Which option you prefer for THREAD+CONDSTORE?

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 13 Aug 2013 18:01:43 +0100

On 12/08/2013 14:32, Jan Kundrát wrote:
> I've just re-read the relevant bits of RFC 4551 again. I have to admit 
> I do not understand the point of including the highest MODSEQ of the 
> matching messages under the MODSEQ key -- what is the use case for 
> that? Sure, it's in the RFC so we cannot change it now, but I'd like 
> to understand the motivation behind this.
This can be used to do partial mailbox resynchronization using 
SEARCH/SORT. So it is like HIGHESTMODSEQ, just for the matching messages.
> Unless I'm missing something, the example response invents a 
> non-existant "SORTED" ESEARCH return value. RFC 5267 (CONTEXT=...) 
> already defines the ESORT extension and the ESORT command and how it 
> uses the ESEARCH response (some servers were broken and used to return 
> an ESORT response, btw). The result is passed as if it was triggered 
> via the ESEARCH command, only in a sorted form, so the correct return 
> data are, IMHO:
>

Ok, I forgot about this :-).

> response claims that the highest MODSEQ of any matching messages is 175.
Yeah, I didn't pay attention to what I was typing. Assume it was 
something like 1175.
> introduced retroacitvely, right?
Yes, this would require an ABNF change. We can do that, as long as the 
new syntax is enabled when a compliant client signals a server by using 
an extension element.
> contains a SHOULD saying that there should only be a SEARCH *or* an 
> ESEARCH response, not both.
I don't think the prohibition intended to apply in this case. One 
advantage of #3 is that it shouldn't break older clients, as long as 
they ignore unrecognized response (which they should).
>> I personally have a slight preference for choices #3/#1, as I don't 
>> want to prolifirate new variants of responses which are not ESEARCH 
>> based.
>
> Indeed, and for the reasons stated above, I prefer 1) after it's fixed 
> to use ALL instead of SORTED.
Right.

Are you Ok with #2 after seeing Timo's response?
> I would love to have the THREAD return option for ESEARCH; in fact, 
> I've tried to introduce it in draft-kundrat-incthread [1]. It will 
> help Trojita understand the matching between commands and responses 
> and therefore enable better pipelining.
Ok. I think your suggested syntax need some tweaks to comply with RFC 
4466, but otherwise this looks sensible. I will have a more detailed 
look later.

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 13 Aug 2013 22:14:03 +0300

On 13.8.2013, at 19.57, Alexey Melnikov <[email-masked]> wrote:

> Right. For example RESMODSEQ (it has the same semantics as MODSEQ SEARCH/SORT item, but causes ESEARCH response to be returned).

I was thinking ESORT compatibility, but then of course ESORT would have to be implemented also by the server. But if we're talking about using ESEARCH anyway, it should be a minor job for servers to add support for ESORT. This actually already works:

a SORT RETURN (ALL) (DATE) UTF-8 MODSEQ 1234
* ESEARCH (TAG "a") ALL 5,3,2,1 MODSEQ 1236

> Ok. Which option you prefer for THREAD+CONDSTORE?

Something similar to the ESORT example I think (so #1 reply with some parameter to switch on ESEARCH reply mode).

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 21 Aug 2013 15:04:29 +0100

On 13/08/2013 20:14, Timo Sirainen wrote:
> On 13.8.2013, at 19.57, Alexey Melnikov<[email-masked]>  wrote:
Ok. I've added this example to the draft.

> Something similar to the ESORT example I think (so #1 reply with some parameter to switch on ESEARCH reply mode).
For now I added sentence that THREAD+CONDSTORE don't return MODSEQ for 
matching messages, as at least 2 server implementations didn't implement 
this (yours and mine). If people object, I can change that.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 21 Aug 2013 15:06:42 +0100

> and therefore enable better pipelining.

I tend to agree with Timo that this needs a new capability. So what you 
are doing in draft-kundrat-incthread (once you fix ABNF as per my other 
comments) is the best way forward.