Subject: [lemonade] SUBMITTED keyword, Outbox and Sent folders, and CATENATE/BURL

From: Peter Coates <[email-masked]>
Date: Thu, 21 Feb 2008 13:50:03 -0800

Within IMAP, the "proper" way to mark a message as draft is to use the
\draft flag.  Similarly the way tentatively to delete a message is to use
the \deleted flag.

Many clients instead use "Drafts" and "Deleted" folders for this purpose.

Other special purpose folders that are commonly used are "Outbox" and
"Sent".  And perhaps "Spam", but I do not want to go there for the moment.
There are no "proper" IMAP ways to mark a message as "Sent" analogous to the
\draft or \deleted flags, still less one meaning queued for submission, or
being submitted.

When I said the "Drafts", "Deleted", "Outbox", and "Sent" are commonly used
folders, I overstate.  Special folders are used for these four functions,
but the names vary.  This can and does cause both interoperability issues
between clients, and confusion on the part of users. 

Actually, the "Outbox" folder has historically not caused issues for IMAP
servers as this has almost always been a folder that was local to the
client, and it is obvious why.  With CATENATE and BURL the outbox function,
or perhaps I should say the outbox state moves to the server.

Let me go through the life cycle of a message being sent from a couple of
putative lemonade compliant client which have chosen two different models.
The first model is to mark the messages to show state, the other uses
different folders for this purpose.  I will then start a discussion about
what a client should do to achieve the best possible interoperability, and
what needs defining for interoperability.

Client model 1 (messages marked with state)
Define three flags (keywords) for the state of outbound messages: 
       QUEUED (to be submitted as soon as possible)
       SUBMITTED (successfully submitted to the submit server)
       
1. The client constructs a message to be sent, eg
C: 1 APPEND INBOX (QUEUED) CATENATE (.....)
S: 1 OK [APPENDUID 1192256 321] done

2. construct a URL for this message, eg
C: 2 GENURLAUTH
IMAP://[email-masked]/INBOX;UIDVALIDITY=1192256/;UID=321;UTLAUTH=SUB
MIT+userid INTERNAL
S: *
IMAP://[email-masked]/INBOX;UIDVALIDITY=1192256/;UID=321;URLAUTH=SUB
MIT+userid:internal:12345678
S: 2 OK done

3. submit the message
c: mail from: <[email-masked]>
s: 250 2.5.0 OK
c: rcpt to: <[email-masked]>
s: 250 2.5.0 OK
c: burl
IMAP://[email-masked]/INBOX;UIDVALIDITY=1192256/;UID=321;URLAUTH=SUB
MIT+userid:internal:12345678 last
s: 250 2.5.0 OK

4. mark the message as having been submitted
C: 4 UID STORE 321 flags SUBMITTED

<done>

Next, the equivalent using Outbox and Sent folders:

1. The client constructs a message to be sent, eg
C: 1 APPEND Outbox CATENATE (.....)
S: 1 OK [APPENDUID 1192256 321] done

2. construct a URL for this message, eg
C: 2 GENURLAUTH
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid INTERNAL
S: *
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid:internal:12345678
S: 2 OK done

3. submit the message
c: mail from: <[email-masked]>
s: 250 2.5.0 OK
c: rcpt to: <[email-masked]>
s: 250 2.5.0 OK
c: burl
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid:internal:12345678 last
s: 250 2.5.0 OK

4. Move the message to the Sent folder
C: 4A SELECT Outbox
C: 4B UID COPY 321 Sent
S: 4B OK [APPENDUID 1418141 198] done
C: 4C UID STORE 321 +flags \deleted
C: 4D UID EXPUNGE 321

<done>

I believe that users will be more comfortable with the second approach than
the former, but that the former is more in the spirit of IMAP than the
latter.  There is nothing to stop a client using a hybrid approach:

1. The client constructs a message to be sent, eg
C: 1 APPEND Outbox (QUEUED) CATENATE (.....)
S: 1 OK [APPENDUID 1192256 321] done

2. construct a URL for this message, eg
C: 2 GENURLAUTH
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid INTERNAL
S: *
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid:internal:12345678
S: 2 OK done

3. submit the message
c: mail from: <[email-masked]>
s: 250 2.5.0 OK
c: rcpt to: <[email-masked]>
s: 250 2.5.0 OK
c: burl
IMAP://[email-masked]/Outbox;UIDVALIDITY=1192256/;UID=321;URLAUTH=SU
BMIT+userid:internal:12345678 last
s: 250 2.5.0 OK

4. Move the message to the Sent folder
C: 4A SELECT Outbox
C: 4B UID COPY 321 Sent
S: 4B OK [APPENDUID 1418141 198] done
C: 4C UID STORE 321 +flags \deleted
C: 4D UID EXPUNGE 321
C: 4E SELECT Sent
C: 4F UID STORE 198 flags SUBMITTED

<done>

The Obvious down side to both the approaches using multiple folders is that
it requires extra selects which are relatively expensive commands and which
lose any CONTEXTS that might be active for the user, and which change the
meaning of any NOTIFY command that might be outstanding.  One solution to
that might be to use a second IMAP session for manipulating the message that
is being submitted.

These has been some discussion as to whether a third state, a SUBMITTING
keyword, is desirable.  This all hinges on recovery after a crash, and
avoiding having multiple clients all submitting queued messages at the same
time.  It is asserted that a combination of the SUBMITTING keyword and the
CONDSTORE semantics can be used to avoid that scenario.

I am not convinced by having a shared Outbox folder being used by multiple
clients.  I think this might lead to surprises for the user.  It might be
better for each client to construct a unique name for its outbox.  On the
other hand, having several Send folders is a known problem.  It would be
nice if we could suggest that if a client is going to use a folder for sent
messages that it use the folder "Sent".  That does not preclude the client
displaying it as whatever it wants, much as INBOX is displayed as various
things in languages other than English.

Peter

_______________________________________________
lemonade mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Fri, 22 Feb 2008 12:00:03 +0000

> C: 4C UID STORE 321 +flags \deleted
There are various failure states here which can be problematic.

Instead of COPY, I'd suggest APPEND Sent (new flags) CATENATE (URL  
"/Outbox;UID=321") anyway - if you're going to change flags, you  
might as well rewrite them.

> C: 4F UID STORE 198 flags SUBMITTED

Even more so here, of course.

> CONDSTORE semantics can be used to avoid that scenario.

I didn't do this - have a "I intend to submit this message now" flag  
- for various reasons, including the simple fact that BCC semantics  
would be somewhat tricky. Also, I'm unnerved by the concept of any  
message submission happening without the user's explicit go-ahead.

With a Queued, Submitted, and Submitting flag, you end up with these  
states:

___	Inbound normal message
__S	Inbound message being resubmitted
_S_	Inbound message resubmitted
_SS	Inbound message being re-resubmitted
Q__	Complete and ready to send
Q_S	Complete message being submitted
QS_	Sent message
QSS	Sent message that is now being submitted

As a client author, the idea of figuring out how to present 8  
different states for a message makes me shudder a bit, frankly, and  
given that there are, here, several potential failure states (for  
example, you successfully submit a message, but find yourself unable  
to add or remove one of the flags, leaving the message is QSS or Q__  
states, one of which means it needs to be submitted, and one of which  
means it is still being resubmitted.)

I'm less than convinced all these are useful, too - the user  
generally will recall if a message was actually sent, they'll also  
know if they meant to BCC it somewhere, etc.

> better for each client to construct a unique name for its outbox.

Yikes. I'm not convinced - I see no reason why Outbox need not be a  
virtual view.

>   On the
I find it considerably more convenient to store sent messages within  
the mailbox I'm storing the related messages in. For example, I might  
store this message (the one I'm composing now) in a "Mailing  
Lists/IETF/Lemonade" mailbox, which the MUA can do because it knows  
the message I'm replying to.

(As it happens, I don't have such a mailbox - it's all in my INBOX).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
lemonade mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Fri, 22 Feb 2008 12:00:12 +0000

In the light of Peter's comments, I'd like to outline how Polymer is  
currently handling similar functionality.

Polymer doesn't use special purpose folders, although it can do.  
Instead its approach is based on using the INBOX primarily.

It's sending pattern is, roughly:

1) If it is helpful to stage the message on the server, do so, using  
tag APPEND (\Draft) ... - for reasons of my own bad programming, it  
will do this even when the message has already been saved - ie,  
multiple sending attempts cause multiple copies of the message, which  
is inarguably wrong. (It creates the new copies quite efficiently,  
using a copy-via-catenate construct, but still).

2) It then creates a signed URL, and submits the message via BURL.

3) Once it receives the final 2xx from the BURL, it marks the message  
with a $submitted flag that I vaguely intended mentioning at some  
point.

So \Draft's meaning is essentially unchanged - "this message has been  
created by the MUA, and may be incomplete". $submitted's meaning is  
obvious.

This yields a state table for messages of something like:

\Draft	$Submitted		State
0		0				Inbound message.
1		0				Message requires further action.
0		1				Message has been resubmitted.
1		1				Message has been "sent".

What I actually do, given this, is to set a viewing filter on DRAFT  
UNKEYWORD $Submitted to colour such messages in violent red, telling  
me that either the submission failed, or hasn't been attempted yet.  
Either way, I probably want to do something about it. (Like, finish  
editing, or attempt to submit it, or something).

Given this, my comments (in another message) which pick on Peter's  
points may be more understandable.

Dave.