Subject: [imap5] Should unsolicited EXPUNGE responses be returned during UID MOVE?

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 13:20:26 -0400

Hi Timo,
(Changing thread since we are now into a technical issue rather than 
charter)

--On May 30, 2012 7:59:19 PM +0300 Timo Sirainen <[email-masked]> wrote:

>> So Arnt tells me.
>
> If I can implement it this way without violating the spec:
>
> 1. Verify that ACLs allow expunge, fail if not
> 2. Do atomic UID COPY without enforcing quota limits
> 3. Expunge the messages
> [4. If expunge for some strange reason fails now, there are probably
> duplicates now. I'm not sure if I want to bother fixing that situation.
> It should "never" happen anyway.]
>
> then I could add it to Dovecot pretty much immediately.

So if the expunge does fail, how are you going to report that to the 
client? Arnt's current spec does not show any * N EXPUNGE responses coming 
back, so I am assuming that the client is supposed to implicitly assume the 
expunges took place, which would cause a problem if they did not take 
place. But that seems wrong to me - shouldn't UID MOVE cause * N EXPUNGE to 
be sent for each message that was actually moved?

-- 
Cyrus Daboo

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 30 May 2012 20:27:22 +0300

On 30.5.2012, at 20.20, Cyrus Daboo wrote:

> So if the expunge does fail, how are you going to report that to the client? Arnt's current spec does not show any * N EXPUNGE responses coming back, so I am assuming that the client is supposed to implicitly assume the expunges took place, which would cause a problem if they did not take place. But that seems wrong to me - shouldn't UID MOVE cause * N EXPUNGE to be sent for each message that was actually moved?

I assumed that this was a bug in the example and was going to report it as some point. I think the EXPUNGE/VANISHED replies should be sent in any case, even if MOVE was required to be fully atomic. Otherwise if the UID MOVE command sends an untagged FETCH/EXPUNGE reply it wouldn't be obvious if the sequence number referred to the state before or after the MOVE expunges.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 30 May 2012 18:56:14 +0100

On 30/05/2012 18:20, Cyrus Daboo wrote:
> client? Arnt's current spec does not show any * N EXPUNGE responses 
> coming back, so I am assuming that the client is supposed to 
> implicitly assume the expunges took place, which would cause a problem 
> if they did not take place. But that seems wrong to me - shouldn't UID 
> MOVE cause * N EXPUNGE to be sent for each message that was actually 
> moved?

Yes, I think lack of EXPUNGE responses is an error for reasons stated by 
Timo.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 30 May 2012 21:11:12 +0000

what about using the sequences in the final tagged response as an 
indication of what was actually successfully moved..  the ones that map 
source messages to new UIDs.
  
We suppress EXPUNGE responses during MOVE already.  When you're moving 
a large set of messages, this improves things quite a bit in terms of 
bw and speed.

----
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

------ Original Message ------
From: "Timo Sirainen" <[email-masked]>
To: "Cyrus Daboo" <[email-masked]>
Cc: "Discussion on drastically slimming-down IMAP." <[email-masked]>
Sent: 31/05/2012 5:27:22 a.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>_______________________________________________
>imap5 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imap5
>
>

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 31 May 2012 00:32:58 +0300

It's doable of course, but I don't think it's worth the extra complexity for 1) the protocol specs and 2) client implementations. The bandwidth savings are pretty minimal.

On 31.5.2012, at 0.11, Adrien W. de Croy wrote:

> what about using the sequences in the final tagged response as an indication of what was actually successfully moved..  the ones that map source messages to new UIDs.
> We suppress EXPUNGE responses during MOVE already.  When you're moving a large set of messages, this improves things quite a bit in terms of bw and speed.
> 
> ----
> Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned during UID MOVE?

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 30 May 2012 23:07:29 +0000

------ Original Message ------
From: "Timo Sirainen" <[email-masked]>
To: "Adrien W. de Croy" <[email-masked]>
Cc: "Cyrus Daboo" <[email-masked]>;"Discussion on drastically 
slimming-down IMAP." <[email-masked]>
Sent: 31/05/2012 9:32:58 a.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>It's doable of course, but I don't think it's worth the extra complexity for 1) the protocol specs and 2) client implementations. The bandwidth savings are pretty minimal.
  
I thought it was already there.

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 31 May 2012 02:35:03 +0300

> I thought it was already there.

The COPYUIDs may or may not be there depending on UIDPLUS extension, but what isn't there is the requirement for clients to parse the UIDs and apply them internally as equivalent to EXPUNGEd UIDs. I see tons of problems with that. Probably nothing that couldn't be overcome by adding a lot of restrictions to the spec and to the client behavior, but it's really not worth the extra complexity.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 31 May 2012 00:05:25 +0000

problem is that to convert back to msgno for an expunge response 
requires a lock on the source.
  
COMPRESS can take the pain out of the actual protocol transmission.
  
Maybe require VANISHED responses instead.

------ Original Message ------
From: "Timo Sirainen" <[email-masked]>
To: "Adrien W. de Croy" <[email-masked]>
Cc: "Discussion on drastically slimming-down IMAP." <[email-masked]>
Sent: 31/05/2012 11:35:03 a.m.
Subject: Re: Re[2]: [imap5] Should unsolicited EXPUNGE responses be 
returned during UID MOVE?

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 31 May 2012 10:42:59 +0200

> requires a lock on the source.

Depends if you cache them or not.  Since msgno is per-connection, it
doesn't matter if they've changed in other sessions meanwhile.
 
> Maybe require VANISHED responses instead.

If you want VANISHED responses you can ENABLE QRESYNC.  I see no reason
not to spit regular EXPUNGED responses otherwise.  The clients that know
how to handle VANISHED can just turn on QRESYNC, the others won't have to
deal with it.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Jan Kundrát <[email-masked]>
Date: Thu, 31 May 2012 11:41:44 +0200

> how to handle VANISHED can just turn on QRESYNC, the others won't have to

As a client writer, I'd expect the UID MOVE to return either explicit
EXPUNGE or explicit VANISHED messages. I do see that omitting that leads
to slightly bigger bandwidth usage (easily mitigated by VANISHED), but
such a behavior would surprise me, based on the way how the rest of the
IMAP works.

In my opinion, section 3 of the draft shall read:

3. UID MOVE Command

***** I'm adding explicit info about the responses here *****
   Arguments:  message set, mailbox name

   Data:       untagged responses: EXPUNGE or VANISHED

   Result:     OK - at least one message has been moved
               NO - none of the messages were moved
               BAD - command unknown or arguments invalid

***** The following remains the same from the 01 draft *****

The UID MOVE command takes two arguments: a set of UIDs and a named
mailbox. It moves each message indicated by UID set to the named mailbox.

The UID MOVE command is the same as a sequence of UID COPY, UID STORE
+FLAGS \DELETED and UID EXPUNGE, with two added requirements:

First, each message SHOULD either be moved or unaffected. The server
SHOULD NOT leave a message in neither or both mailboxes afterwards (even
if the server returns a tagged NO response).

Second, the messages MUST NOT have the \Deleted flag set in the target
mailbox.

***** And here are some additions ******

If at least one message was successfully moved to the target mailbox,
the command MUST report success with a tagged OK response. If none of
the messages could be moved, the server MUST respond with a tagged NO.

Clients are informed about the fate of the individual messages through
regular untagged EXPUNGE responses, as defined in RFC 3501. In case the
current session has enabled the QRESYNC extension, untagged VANISHED
shall be send instead of the EXPUNGE responses.

***** The example is also modified with EXPUNGEs and the number of
messages is reduced *******

An example:
     C: a UID MOVE 42:45 forble
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: a OK [COPYUID 432432 1202:1205] Done

****** End ******

I see a few issues to be discussed, though:

- There's a requirement that a message in the target mailbox must not
have the \Deleted flag. What is supposed to happen when I UID MOVE a
message which was already marked as \Deleted before?

- Sending the untagged EXPUNGE/VANISHED and only after them the COPYUID
might be troublesome for some clients -- for example, my code will react
to EXPUNGE by removing the cached data for that message from the on-disk
cache. When the COPYUID arrives, the data is already gone. (My code
actually ignores COPYUID for now anyway, but that's not relevant here.)

- Postponing of the untagged EXPUNGE/VANISHED until after the tagged OK
is probably a bad idea, so what about a special "VANISHED (MOVED)" which
would hint the client that the data shall not be purged immediately?

- Section 4.3 shall probably say that servers supporting the UIDPLUS
extension SHOULD include the COPYUID response code in the tagged OK, but
that means that the introduction in section 4 shall not claim that there
are no requirements in that section.

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 31 May 2012 12:52:07 +0300

> would hint the client that the data shall not be purged immediately?

Another possibility:

     C: a UID MOVE 42:45 forble
     S: * OK [COPYUID 432432 1202:1205] Messages moved
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: * 15 EXPUNGE
     S: a OK Done

---

From: Jan Kundrát <[email-masked]>
Date: Thu, 31 May 2012 12:11:35 +0200

>      S: * OK [COPYUID 432432 1202:1205] Messages moved

The untagged OK with COPYUID doesn't specify to which UID MOVE it is
related, unfortunately, and will therefore break with concurrent UID
MOVE operations. My GUI will happily send concurrent UID MOVEs when the
connection is slow and user moves her mouse fast enough, so I believe
it's a real problem.

I'm not aware of any extension using multiple response codes at once
(and hence "* OK [TAG a COPYUID 432432 1202:1205] Messages moved" is not
really a COPYUID anymore), what about a special MOVEUID response code?

* OK [MOVEUID a 432432 1202:1205] Messages moved

I like that, and it also means that my proposed "VANISHED (MOVED)" is
not necessary.

And to reduce the number of possible combinations, what about making the
MOVEUID a mandatory response, no matter whether UIDPLUS is supported?

Cheers,
Jan

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 31 May 2012 10:42:20 +0000

well one thing to think about.
  
Since there are already servers and clients deployed, unless we allow 
there to be some way to distinguish between a MOVE extension that sends 
EXPUNGES and one (legacy) that doesn't, do we need to call this MOVE2?
  
Adrien

----

------ Original Message ------
From: "Timo Sirainen" <[email-masked]>
To: "Jan Kundrát" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 31/05/2012 9:52:07 p.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 31 May 2012 14:12:21 +0200

On 05/30/2012 07:20 PM, Cyrus Daboo wrote:
> Arnt's current spec does not show any * N EXPUNGE responses coming back,
> so I am assuming that the client is supposed to implicitly assume the
> place. But that seems wrong to me - shouldn't UID MOVE cause * N EXPUNGE
> to be sent for each message that was actually moved?

My FUBAR. I'll correct and repost.

(My code might actually send EXPUNGE a few instants later. I think 
that's okay, so long as both server and client change their MSN maps in 
sync.)

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 31 May 2012 20:58:11 +0000

we do need to think whether we're creating a compatibility issue with 
deployed systems.
  
For instance, if we can't tell whether a client will break on MOVE if 
we send EXPUNGEs or not, we'd need to basically stop advertising MOVE
  
which completely defeats the purpose.
  
Also.  Even though we have to send EXPUNGEs to any other connected 
client on that mailbox (that didn't do the move), the approach is a bit 
wasteful of bandwidth etc.
  
If you issue a command, you want to know when it's done and how it 
turned out.  If the MOVE completes in its entirety without issue 
(99.99% of the time), then  a simple "yeah it worked" response is the 
most efficient.  We can inform of exceptions, e.g. "something broke", 
or "these messages didn't work".  In the end, if there's a failure so 
what if the client has to resynch indices.
  
But loading the protocol success scenario with major bloat is just that 
- bloat.
  
the client said "move these messages".  If the server says "I did it", 
the client can safely act as if the server did it, and remove from 
local.  EXPUNGE responses are completely redundant.
  
Adrien
----

------ Original Message ------
From: "Arnt Gulbrandsen" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 1/06/2012 12:12:21 a.m.
Subject: Re: [imap5] Should unsolicited EXPUNGE responses be returned 
during UID MOVE?
>that's okay, so long as both server and client change their MSN maps 
>in sync.) 
>
>Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 1 Jun 2012 08:03:26 +0200

Listen closely. The missing EXPUNGEs are a document bug, not 
representative of any servers I know.

Arnt