Subject: [imap5] Proposal for "imapmove" working group

From: Barry Leiba <[email-masked]>
Date: Wed, 30 May 2012 09:39:18 -0400

At the instigation of some proponents, I am bringing the attached
working group charter proposal to the IESG.  Short version: the
proposal is to develop a Proposed Standard for an IMAP MOVE protocol
extension, which will most likely support only UID MOVE, and to do it
on a very short schedule.  An initial proposal for such a protocol is
here:
   http://datatracker.ietf.org/doc/draft-gulbrandsen-imap-move/
...and that's a likely starting point for the working group.

Again: a very short schedule.  We have a good starting proposal and a
fairly simple task (but see below), along with several implementations
that are asking for this.  The discussion and progress will be tightly
managed.

Note that this proposal comes with some controversy -- mostly, but not
entirely, based on opinions that it's not really needed nor necessary,
and that it's not as easy to get it right as some think.  See the
(long) thread of discussion from the imap-protocol mailing list that
starts here:
http://mailman2.u.washington.edu/pipermail/imap-protocol/2010-June/001088.html

Comments on the charter proposal are welcome, and should be posted to
this list, <[email-masked]> (and no need to CC me).  The charter will
be on the 7 June IESG telechat for initial review, and will be back on
21 June for possible approval.

Barry Leiba, Applications AD

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 30 May 2012 17:20:15 +0300

> 21 June for possible approval.

I like how UID MOVE is defined to be a combination of UID COPY+STORE+EXPUNGE commands. That's why I think the word "atomic" should be removed from everywhere. The command is useful even if it's not strictly atomic, just as long as it normally looks that way to clients.

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 10:39:51 -0400

Hi Barry,

> that are asking for this.  The discussion and progress will be tightly

I seriously question the need to spin up a working group just for this one 
proposal. I realize that there is significant controversy over the need for 
this extension, but having a working group won't change that fact.

In my opinion this should be handled as an individual submission but with 
an "experienced" shepherd "appointed" by the Apps ADs to manage the 
development and consensus of this proposal - perhaps on a dedicated mailing 
list, but maybe reusing imapext or some other suitable list. I would rather 
not waste time looking over a charter and milestones and all the rest of 
the WG setup overhead when there is just one document to deal with.

In any case, I am willing to volunteer to chair or be the shepherd as I 
think this extension is important and long overdue.

-- 
Cyrus Daboo

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 30 May 2012 15:45:45 +0100

On 30/05/2012 15:39, Cyrus Daboo wrote:
> Hi Barry,
>
> --On May 30, 2012 9:39:18 AM -0400 Barry Leiba 
> <[email-masked]> wrote:
>
> I seriously question the need to spin up a working group just for this 
> one proposal. I realize that there is significant controversy over the 
> need for this extension, but having a working group won't change that 
> fact.
>
> In my opinion this should be handled as an individual submission but 
> with an "experienced" shepherd "appointed" by the Apps ADs to manage 
> the development and consensus of this proposal

Cyrus, there was a private round of discussion with ADs and they 
indicated that they are not willing to process this document as AD 
sponsored (rightly so in this case).

> - perhaps on a dedicated mailing list, but maybe reusing imapext or 
> some other suitable list. I would rather not waste time looking over a 
> charter and milestones and all the rest of the WG setup overhead when 
> there is just one document to deal with.
>
> In any case, I am willing to volunteer to chair or be the shepherd as 
> I think this extension is important and long overdue.
>

---

From: Barry Leiba <[email-masked]>
Date: Wed, 30 May 2012 10:45:58 -0400

Cyrus says...
> this extension, but having a working group won't change that fact.

Well, the IESG discussed this informally, and decided that a working
group is the best approach.  At some level, we ought never need a
working group for most things: WEIRDS, REPUTE, OAUTH, ALTO....

This one is being done in the manner that the RAI area has been
handling many of its working groups: short-lived, focused groups, and
many of them.  You'll note that the total time between the message I
posted this morning and the chartering of the working group is likely
to be three weeks.  That's not a lot of difficulty involved with
spinning up a working group.

Barry

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 11:39:13 -0400

Hi Barry,

> spinning up a working group.

Well, OK. If that seems to be what the IESG wants to do, then fine. In that 
case I will move on to commenting on the charter...

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 12:22:09 -0400

Hi Barry,
Comments on charter below.

--On May 30, 2012 9:39:18 AM -0400 Barry Leiba <[email-masked]> 
wrote:

> Mailing Lists:
>   General Discussion: [email-masked]
>   To Subscribe:       https://www.ietf.org/mailman/listinfo/imap5
>   Archive:            http://www.ietf.org/mail-archive/web/imap5/

Which mailing list? imapext or imap5?

> It's often the case that an IMAP client needs to move (not copy)
> messages from one mailbox to another.  The mechanism that IMAP
> provides to do that is a multi-step process:
> 1. Copy the messages from the source mailbox to the target mailbox.
> 2. Flag the original messages in the source mailbox as deleted.
> 3. Expunge the deleted messages from the source mailbox.
>
> Implementors have long pointed out some shortcomings with this
> approach.  Because the moving of a message is not an atomic process,
> interruptions can leave messages in intermediate states.  Because
> multiple clients can be accessing the mailboxes at the same time,
> clients can see messages in intermediate states even without
> interruptions.  If the source mailbox contains other messages that are
> flagged for deletion, the third step has the side effect of expunging

                                       ^^^

Really it is "can have the side effect" since clients might have the option 
of using UID EXPUNGE, or could play tricks of undeleting the non-copied 
deleted messages, doing the expunge, then re-deleting - of course no one in 
their right mind would do that...

> more than just the set of moved messages.  And servers with certain
> types of back-end message stores might have efficient ways of moving
> messages, which don't involve actual copying of data.  Such
> efficiencies are often not available to the copy/flag/expunge process.

> The IMAP MOVE extension (imapmove) working group has the single task
> of developing an atomic IMAP MOVE command that will move a set of

                   ^^^^^^           ^^^^^^^

Use "extension" instead of "command" as the actual command is "UID MOVE" as 
per Arnt's spec. As per Timo, I am not sure we should state it will be 
"atomic" as that may restrict what we do in the spec.

> messages from a source mailbox to a target mailbox in a single
> operation.  The group will use draft-gulbrandsen-imap-move as a
> starting point, and will produce a Standards Track document.

> As part of the protocol development, implementation experience on both
> the client and server side is highly desireable, so that the actual
> operational value of this extension can be assessed. The working group
> will document the results of this experience on the working group
> wiki.

Are there existing client and server implementations of the 
draft-gulbrandsen-imap-move right now? Does "implementation" also cover 
actual interoperability testing, or does it only speak to the actual 
ability to modify existing servers/clients to adopt the extension?

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 12:32:18 -0400

Hi Barry, [email-masked],

--On May 30, 2012 12:22:09 PM -0400 Cyrus Daboo <[email-masked]> wrote:

> Use "extension" instead of "command" as the actual command is "UID MOVE"
> as per Arnt's spec. As per Timo, I am not sure we should state it will be

Thinking some more, perhaps better text would be:

... developing an IMAP MOVE extension that defines a single command to move 
a set of ...

i.e., "single command" rather than "atomic".

-- 
Cyrus Daboo

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 30 May 2012 18:45:30 +0200

On Wed, May 30, 2012 at 12:22:09PM -0400, Cyrus Daboo wrote:
> draft-gulbrandsen-imap-move right now? Does "implementation" also
> cover actual interoperability testing, or does it only speak to the
> actual ability to modify existing servers/clients to adopt the
> extension?

Cyrus git-master has XMOVE implemented.  We use it in FastMail for
our web interface to avoid quota issues when moving messages via
the web interface rather than the old hack of working out the quota,
then temporarily raising the limit, doing the copy+expunge and then
reducing the limit again.

Bron.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 30 May 2012 12:47:41 -0400

> Which mailing list? imapext or imap5?

Typo (or braino).  There is no "imapext" list.

> of using UID EXPUNGE

But UID EXPUNGE is part of the UIDPLUS extension, not in base IMAP
(and I'm not sure how widely deployed UIDPLUS is.  And, as you say,
one would have to be mentally unstable to try to work around this the
other way.  Still, "can have" is fine, and I've changed the text.

>> of developing an atomic IMAP MOVE command that will move a set of

It's meant to be "atomic" from the client's view, whether or not it's
atomic in the server.  But I prefer your later suggestion of "single
command", so I've changed it accordingly.

> draft-gulbrandsen-imap-move right now?

So Arnt tells me.

> Does "implementation" also cover
> actual interoperability testing, or does it only speak to the actual ability
> to modify existing servers/clients to adopt the extension?

If there's interop testing, it would be very fine to document that
here.  But that's not what this is about; this is to show that there's
a real need for this, and active work on implementations (as opposed
to our developing another IMAP extension that gets essentially no
use).

Attached is the latest charter version, with these changes.

Barry

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 30 May 2012 19:59:19 +0300

>> draft-gulbrandsen-imap-move right now?

If I can implement it this way without violating the spec:

1. Verify that ACLs allow expunge, fail if not
2. Do atomic UID COPY without enforcing quota limits
3. Expunge the messages
[4. If expunge for some strange reason fails now, there are probably duplicates now. I'm not sure if I want to bother fixing that situation. It should "never" happen anyway.]

then I could add it to Dovecot pretty much immediately.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 30 May 2012 18:04:23 +0100

On 30/05/2012 17:47, Barry Leiba wrote:
> On Wed, May 30, 2012 at 12:22 PM, Cyrus Daboo<[email-masked]>  wrote:
  [...]
> (and I'm not sure how widely deployed UIDPLUS is.

A side note: it is very widely deployed. As far as I am concerned, if a 
server doesn't implement UIDPLUS, it is pretty much unusable and 
unlikely to work with a general purpose IMAP client.

>    And, as you say,

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 30 May 2012 13:11:19 -0400

Hi Barry,

> Typo (or braino).  There is no "imapext" list.

Well there is, just not quite what you wrote:

List-Archive: <http://www.imc.org/ietf-imapext/mail-archive/>
List-ID: <ietf-imapext.imc.org>

That list is still active to some extent and might be more appropriate for 
an extension to IMAP rather than "IMAP 5", but I am not bothered either way.

>> Really it is "can have the side effect" since clients might have the
>> option of using UID EXPUNGE
>

Hehe - well I guess I must have been mentally unstable at some point in 
time because I did actually implement the "undelete, sequence expunge, 
redelete" behavior in Mulberry to handle the case of a non UIDPLUS server - 
however that is only used when playing back disconnected operations where 
an expunge was explicitly triggered by the user whilst offline. In Mulberry 
the "move" operation simply does copy + mark deleted - it does not expunge 
- the user is on the hook to explicitly do that.

-- 
Cyrus Daboo

---

From: Barry Leiba <[email-masked]>
Date: Wed, 30 May 2012 13:24:07 -0400

> List-ID: <ietf-imapext.imc.org>

Right; that's not an "imapext" list, it's an "ietf-imapext" list.  And
it doesn't live at ietf.org.  So, no, not what I meant at all.

The IESG very strongly wants to use mailing lists at ietf.org, unless
there's a very good reason not to.  We could create a new
<[email-masked]> list, or use the not-very-active imap5 list for a
few months.  The latter seemed better to me, since many of the people
who will want to follow this are already there.

I'm happy to instead create a new list, if consensus goes that way.

Barry

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 30 May 2012 22:43:35 +0200

> ability to modify existing servers/clients to adopt the extension?

There are several servers which do what the document describes.

I know of several "clients" too, but perhaps not as you'd see it. 
Technically a perl script that classifies and moves mail is a client, right?

https://jira.zarafa.com/browse/ZCP-6695 may be interesting to you. No, 
Zarafa has nothing to do with AOL.

Arnt

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Wed, 30 May 2012 21:07:28 +0000

------ Original Message ------
From: "Cyrus Daboo" <[email-masked]>
To: "Barry Leiba" <[email-masked]>;"[email-masked]" 
<[email-masked]>
>actual ability to modify existing servers/clients to adopt the 
  
WinGate has had it for a while now.
  
We noticed speed improvements using it with Thunderbird for mail 
filtering.  
  
Our server has a naive (aka deep) COPY using Maildir style.  We still 
support windows 2000 which doesn't support CreateHardLink.  Scheduled 
to drop support for 2k soon though.  Will definitely be looking to 
improve COPY with this.
  
The current mail client I use doesn't do MOVE though. It is slightly 
slower on filtering for that reason, esp on higher latency connections 
to the server (more R-Ts).
  
We don't do nested locks either, although it's feasible to do lock 
checks prior to locking to error back to the client rather than 
deadlock (classic 2 thread deadlock issue).

We also support MOVE (without UID).  It resolves source msgids to UIDs 
inside a lock of the source before effectively doing a MOVE.  This is 
probably error-prone in some pathological border cases, but I think the 
effect at worst would be a resynch in the interfering client.
  
Adrien
  
>
>
>
>-- Cyrus Daboo 
>
>_______________________________________________ 
>imap5 mailing list 
>[email-masked] 
>https://www.ietf.org/mailman/listinfo/imap5

---

From: Jan Kundrát <[email-masked]>
Date: Wed, 30 May 2012 23:32:22 +0200

> reducing the limit again.

I've just added support for UID XMOVE into Trojitá; it works fine
against FastMail's public servers.

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 31 May 2012 00:00:52 +0200

Earlier tonight I posted a link to a bug report for zarafa and thunderbird, showing problems with a proprietary aol extension they both implement.

If that does not show real need, I don't know what would suffice.

Barry Leiba <[email-masked]> wrote:

>ability
>------------------------------------------------------------------------
>

Arnt

---

From: Barry Leiba <[email-masked]>
Date: Thu, 31 May 2012 10:10:50 -0400

> I'm happy to instead create a new list, if consensus goes that way.

After some off-list discussion, I think we WILL create a new list,
<[email-masked]>.  If anyone *objects* to this, please post here
and say why.  Otherwise, the next version of the charter proposal will
say that.

Barry

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 31 May 2012 17:18:29 +0300

> and say why.  Otherwise, the next version of the charter proposal will

I'm not highly against it, but I think we already have too many
imap-related mailing lists. Gets difficult to know sometimes where to
post a question and if all the important people are reading the right
lists.

---

From: Barry Leiba <[email-masked]>
Date: Thu, 31 May 2012 10:50:52 -0400

> post a question and if all the important people are reading the right

ACK.
That was my thinking in wanting to do this here (on imap5).  But
because this list was chartered for a different purpose, there's
(reasonable) concern that there will be distractions in both
directions if we temporarily hijack it for this one.

I will probably propose to the IESG that we accept this as an
exception, and use the <[email-masked]> list that's already
there.  That fits better in its purpose, but puts a new WG mailing
list off of IETF premises.  The IESG has stopped short of saying that
should never happen, but has a strong preference against it.  So I
want everyone to be prepared in case we choose to create a new list.

Barry

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 31 May 2012 10:59:08 -0400

Hi Barry,

> want everyone to be prepared in case we choose to create a new list.

Paul at IMC has transferred a number of lists over to IETF control. So 
would it not make sense to create [email-masked] and transfer over 
ietf-imapext membership? Then we have one IETF controlled list going 
forward.

-- 
Cyrus Daboo

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 31 May 2012 18:02:26 +0300

> want everyone to be prepared in case we choose to create a new list.

Would it be possible to create a more generic list for "Any new
IMAP-related workgroups"? I'd even prefer if it was somehow aliased to
morg list (so that they would really be the same list but with two
addresses/names), but I guess that's too much trouble.

---

From: Barry Leiba <[email-masked]>
Date: Thu, 31 May 2012 11:12:57 -0400

> Paul at IMC has transferred a number of lists over to IETF control. So would
> it not make sense to create [email-masked] and transfer over ietf-imapext
> membership? Then we have one IETF controlled list going forward.

That's a good suggestion; I'll talk with PH about that.

Barry

---

From: Tony Hansen <[email-masked]>
Date: Thu, 31 May 2012 18:06:30 -0400

> post a question and if all the important people are reading the right

I was going to attempt some minor humor and suggest using 
[email-masked].

But seriously, Timo has a very valid point.

How about if we look into moving imapext from imc.org to ietf.org, and 
then use [email-masked]?

     Tony

---

From: Tony Hansen <[email-masked]>
Date: Thu, 31 May 2012 18:11:12 -0400

> then use [email-masked]?

Sorry about the duplicate suggestion -- that's what I get for not 
hitting Send right away, and then clearing off my desktop before 
quitting time without looking to see how the discussion has progressed.

     Tony