Subject: [imap5] In response to the Crispinogram.

From: Dave Cridland <[email-masked]>
Date: Fri, 01 Aug 2008 23:29:50 +0100

Answer to Mark, in order:

1) The general idea is to start with a "base" of IMAP4+extensions,  
and then whittle it down. My criteria for retaining items is "What  
active client authors are actually using", others may differ. I'll go  
along with "What active client authors would like to use if it  
works", mind, but if anyone says "You absolutely need feature X, even  
though nobody uses it at present", then I shall be forced to go "All  
Crispin" on them. (See points 7~10).

One key agreement we had was that we wouldn't bother if we couldn't  
get client buy-in.

And yes, I certainly agree that one important prerequisite is  
understanding IMAP properly to begin with. It's always been my  
understanding that IMAP aims to shift complexity to the server, and  
provide flexibility to the client, I don't intend anything which  
prevents that.

2) Agreed. I'd like to see a concrete proposal for replacing  
BODYSTRUCTURE before I'm interested in talking about removing it.

I would, however, go along with the addition of including the part  
specifiers in it - I appreciate it's a little superfluous, but it  
seems to cause endless problems for client authors.

3) Agreed. UTF-8 and '/' delimiter, no other choice. Also, we should  
remove the reference argument to LIST, and, quite possibly, remove  
wildcards. Hence depths of zero, one, and infinity should be the  
golden rule once more.

Finally, ditch NAMESPACE in favour of a fixed, bounded, heirarchy  
root.

4) Agreed. I like BAD. It's exceedingly useful for client developers,  
who should only receive BAD in very rare circumstances, and typically  
will need to handle it as an exception.

5) I'm happy and content with killing LOGIN and PREAUTH. LSUB may as  
well go, everyone uses it in a different way anyway. LIST we keep,  
just massively simplify it - I agree it belongs ideally in a distinct  
protocol, but moving it to one would be more effort than it's worth.

6) I'm not convinced. It's important to retain the property of  
mailbox sequence, certainly, and I'd retain the sequence search key,  
for example, but what I'd be happy to see go is the MSN-as-reference,  
and I disagree that this removes a class of client. Remember, I write  
an online, potentially cacheless, client, and it doesn't use non-UID  
variants of commands, and suffers no real ill effects as a result.  
What it can do is operate seamlessly with and without a cache, using  
the same codebase.

7) Actually, a more useful \Recent can be emulated trivially just by  
setting a keyword on delivery set by, for example, SIEVE, and turned  
off only by those clients actually performing the client-side  
filtering, using CONDSTORE - or simply set once a message has been  
filtered, eliminating special delivery-time handling.

\Recent only works for this if you're using homogeneous clients,  
which isn't all that common if you're using multiple devices, and  
only then if the clients happen to never crash, connectivity never  
fail, and power be always available.

Essentially, I think that \Recent adds complexity, and has been  
superceded by newer developments, allowing stable and reliable  
client-side filtering etc.

A \Recent for mailboxes, however, is a different beast - I'd like to  
fully specify \Marked and \UnMarked.

8) Agreed, no need to mandate SIEVE. Sure it's a nice chunk of  
functionality, but it's orthogonal the the issue of message stores.

9) Oh, *so* in agreement. We were discussing in MORG how we need a  
SEARCH/SORT that basically says "Hey, messages about IMAP. Find 'em."  
We do not need a "Hey, messages containing a four letter word  
prefixed with two digits. Find 'em."

10) I'm not going to bother gracing this with an answer - not because  
I don't think this group won't benefit from a long view, and not  
because I think this group won't benefit from your opinion, either,  
but because I do not believe for a moment that you will refrain from  
giving either when you think it important.

Of course, I did actually grace that with an answer, thus ruining my  
rhetoric, damn it.

Some suggestions of my own, none contentious:

a) Death to modUTF-7!

b) Simplify the syntax.

Response codes should be in (), "phrases" should be quoted strings,  
etc:

A01 NO (BLURDYBLOOP) "This is the phrase, folks"

I'd like to homogenize FETCH specifiers, too.

c) More UTF-8, I say! More!

Quoted strings should be UTF-8. ENVELOPE and BODYSTRUCTURE should be  
doing RFC 2047 decoding to UTF-8. Basically, everything should be  
UTF-8.

d) Tagged intermediates.

Oh yes. Bring 'em on:

C: A01 COMMAND
S: A01 RESULT
S: A01 RESULT
S: * UNRELATED (PUSH NOTIFICATION)
S: A01 OK "Completed, and I remembered about strings-for-phrase."

e) First-class FLAGS for spam handling.

So that all IMAP5 clients mark spam the same way. Hopefully.

I have some otheer ideas, but they require more thought, and possibly  
a bit of code to prove they work at all.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Fri, 1 Aug 2008 18:58:21 -0700

> active client authors are actually using", others may differ.

The problem that you then get into flame wars as to "does this client
count."  If Pine, Alpine, and any other client based upon my c-client
API counts, then very little can be deleted from the base specification;
and the SORT, THREAD, and MULTIAPPEND extensions are needed
for performance over slow links.

> provide flexibility to the client, I don't intend anything which

In that case, you need to preserve server-based message parsing,
searching, and sorting/threading.

> [BODYSTRUCTURE]

The problem in doing that is that you then create a silly state in which
the new BODYSTRUCTURE gives part specifiers that don't actually
correspond to the structure.

That will happen.  Don't think for one minute that it won't.

> Finally, ditch NAMESPACE in favour of a fixed, bounded, heirarchy

Yes.

> LIST we keep,

Massively simplifying means ditching all the LISTEXT stuff too, kicking
it back to the RFC 1176 FIND command.

Oh, you didn't mean that massive...  ;-)

> and I disagree that this removes a class of client.

Pine, Alpine, and every other application based upon the c-client API
uses MSN as reference.  Most application calls that use UIDs translate
immediately into MSN.  The only thing Pine/Alpine use UIDs for is
to identify which message to set \Answered when resuming a draft from
a previous sesson.

Remember too that POP3 uses MSNs.

> client-side filtering etc.

I may be convinced that \Recent has been superceded, but not that
it adds complexity.  It's been in Internet mailboxes from the earliest
days, only has not been called that.  It's the "O" status flag in mbox
format.

Basically, what you are doing is prohibiting IMAP5 from exporting any
sort of pre-existing mail store, and instead you are defining an IMAP5
mail store.  I am not convinced that such an effort will be successful;
the cloud providers seem to be unwilling to accept the IETF defining
the semantics of their store.

Ignore this warning at your peril.

> giving either when you think it important.

If IMAP5 ends up being as bad a train wreck as LEMONADE, there won't
be much need for me to participate.  As in LEMONADE, I will try
initially to stop the train wreck, but once it becomes inevitable I'll cut
my losses.

> A01 NO (BLURDYBLOOP) "This is the phrase, folks"

Do you intend to have responses carrying data?

If not, I fail to see the point.

> d) Tagged intermediates.

Oh no, now we're back to RFC 1203 and its attempt to repudiate the IMAP
data model.  Do not destroy what you do not understand.

If you need to have data tags (SEARCH, SORT, and THREAD being the
obvious examples), introduce data tags.  Do not overload command tags.

Let me repeat: do not overload command tags to be data tags.

There is quite a bit more to that statement than meets the eye.  I suggest
that you think about what that might be before commenting further.

-- Mark --

_________________________________________________________________
With Windows Live for mobile, your contacts travel with you.
http://www.windowslive.com/mobile/overview.html?ocid=TXT_TAGLM_WL_mobile_072008
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dave Crocker <[email-masked]>
Date: Sat, 02 Aug 2008 09:41:13 +0100

Dave Cridland wrote:
> 1) The general idea is to start with a "base" of IMAP4+extensions, and 
> then whittle it down. My criteria for retaining items is "What active 
> client authors are actually using", others may differ. I'll go along 
> with "What active client authors would like to use if it works", mind, 
> but if anyone says "You absolutely need feature X, even though nobody 
> uses it at present", then I shall be forced to go "All Crispin" on them. 

I suggest something more strict:

What is in existing clients, and is known to work properly and well consistently 
across all of them, and is is viewed as being absolutely essential.  This 
creates an extremely tight core.  Given the variability of IMAP implementation 
behaviors, this creates an extremely solid base of existing implementation.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Randall Gellens <[email-masked]>
Date: Sat, 2 Aug 2008 16:30:58 -0700

>  Answer to Mark, in order:

I agree with Dave.  He put it well.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
There's nothing in life so difficult that a Microsoft manual can't
make it completely incomprehensible.                --Douglas Adams
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Randall Gellens <[email-masked]>
Date: Sat, 2 Aug 2008 17:00:51 -0700

>  the semantics of their store.

IMAP has attempted to both allow more-or-less direct export of a 
local mail store (hence the range of server variance in all sorts of 
things, such as hierarchy) and to be an abstraction for access to 
such mail stores.

Reducing the former in favor of a simpler way to go the latter helps; 
I don't see it imposing a new IMAP5 mail store, but rather providing 
a conceptually less complex model of whatever mail store lies 
underneath.

-------------- Randomly-selected tag: ---------------
To err is human, to moo bovine.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Randall Gellens <[email-masked]>
Date: Sat, 2 Aug 2008 17:28:40 -0700

At 9:41 AM +0100 8/2/08, Dave Crocker wrote:

>  What is in existing clients, and is known to work properly and well 
> consistently across all of them, and is is viewed as being 
> absolutely essential.

My fear is that this is the null set.   What part of IMAP does 
everyone get right?

>  This creates an extremely tight core.  Given the variability of 
> IMAP implementation behaviors, this creates an extremely solid base 
> of existing implementation.

-------------- Randomly-selected tag: ---------------
Never precede any demo by a comment more predictive than "Watch this!".
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Sat, 2 Aug 2008 21:34:17 -0700

> a conceptually less complex model of whatever mail store lies

Careful.  That would be an argument to delete "dual-use".

_________________________________________________________________
Your PC, mobile phone, and online services work together like never before.
http://clk.atdmt.com/MRT/go/108587394/direct/01/
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dave Crocker <[email-masked]>
Date: Sun, 03 Aug 2008 12:57:30 +0100

Randall Gellens wrote:
> Reducing the former in favor of a simpler way to go the latter helps; I 
> don't see it imposing a new IMAP5 mail store, but rather providing a 
> conceptually less complex model of whatever mail store lies underneath.

+1

d/
-- 

   Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Sun, 03 Aug 2008 13:11:53 +0100

Randall Gellens wrote:
>> consistently across all of them, and is is viewed as being absolutely 
>> essential.
> 
> My fear is that this is the null set.   What part of IMAP does everyone 
> get right?

Randy, et al,

Certainly a fair concern,

My intent is to suggest a very strict base from which to start, since I've no 
doubt there will be a variety of justifications for expanding to a larger scope.

I take the desired outcome to be something that is more easily and reliably 
possible to implement correctly. The IETF history says that that requires 
*extremely simple* basic capabilities.  It's fine to add extensions to that 
base, but basic interoperability should entail only very basic functionality.

As for some other concerns already raised on the list:

1. It seems unlikely that there is a null set of capabilities, implemented in 
all clients correctly.  If indeed it is null, then a) that says something about 
IMAP that is probably unprecedented in IETF work, and b) relax the rule the 
smallest amount necessary to get something workable. [I leave the criteria for 
'smallest amount' as a group exercise.]

2. Yes, selecting candidate implementations will be a challenge.  Merely one of 
many challenges for this effort.  I suggest looking for the implementations that 
target the mass market, since that will get the broadest benefit.  This clearly 
will short-change the more specialized markets, but isn't that market already 
getting its functionality with IMAP4?  In any event, there simply *must* be a 
means of simplifying the decision process, as well as its output, or it won't 
achieve that simplified output.

3.  "We can't live without feature x" is appealing and dangerous. I suggest 
finding a very simple basis for applying it and filtering, such as massive rough 
consensus that it is needed.  Even then, if there is any organized(*) 
resistance, it is left out.    In other words the decision process should be 
biased against including things.  It is always possible to relax things later, 
as the process continues, but rarely possible to make it strictly.

d/

(*) By "organized" I mean that some diversity of people organize to lobby for 
something.  The point is to require something beyond religious fervor. 
Diversity among multiple people raises the likelihood that the view has 
broad(er) applicability.

-- 

   Dave Crocker

---

From: Randall Gellens <[email-masked]>
Date: Mon, 4 Aug 2008 18:00:06 -0700

At 1:11 PM +0100 8/3/08, Dave Crocker wrote:

>  My intent is to suggest a very strict base from which to start, 
> since I've no doubt there will be a variety of justifications for 
> expanding to a larger scope.
>
>  I take the desired outcome to be something that is more easily and 
> reliably possible to implement correctly. The IETF history says 
> that that requires *extremely simple* basic capabilities.  It's 
> fine to add extensions to that base, but basic interoperability 
> should entail only very basic functionality.

Yes, I agree.

>  I suggest looking for the implementations that target the mass 
> market, since that will get the broadest benefit.  This clearly 
> will short-change the more specialized markets, but isn't that 
> market already getting its functionality with IMAP4?

I wonder if mobile devices are included in the mass market definition?

>  In any event, there simply *must* be a means of simplifying the 
> decision process, as well as its output, or it won't achieve that 
> simplified output.

Yes, I agree.

>
>  3.  "We can't live without feature x" is appealing and dangerous. I 
> suggest finding a very simple basis for applying it and filtering, 
> such as massive rough consensus that it is needed.  Even then, if 
> there is any organized(*) resistance, it is left out.    In other 
> words the decision process should be biased against including 
> things.  It is always possible to relax things later, as the 
> process continues, but rarely possible to make it strictly.

Yes, and also to add extensions later, after we have interoperable base specs.

-------------- Randomly-selected tag: ---------------
Illegal aliens have always been a problem in the United States.
Ask any Indian.                                 --Robert Orben
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Mon, 4 Aug 2008 21:07:13 -0400

> I wonder if mobile devices are included in the mass market definition?

I wonder if there's any point in having two levels of IMAP support,  
like:

1) The basics for offline-clients that prefer to know about all  
messages (similar to Outlook/Thunderbird/Mail.app).

2) A bit more features for online-clients like webmails and mobiles.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 4 Aug 2008 18:19:01 -0700 (PDT)

> I wonder if there's any point in having two levels of IMAP support, like:

To my way of thinking, extensions must die. Otherwise you end up right 
back in the swamp you're trying to crawl out of.

--lyndon

   I think 3B2 code deserves its own place in hell.  Poring over the
   ESS#5 code, someone found that there were lots of strcmp(p, "f(")
   == 0 checks (I may have gotten the exact string wrong but it's
   close).  It took us a while to figure out why.  Apparently, location
   0 on the 3b had the 3 bytes 'f' '(' '\0', someone noticed that when
   programs blew up they were pointing to "f(", and the worlds most
   amazing kludge for detecting nil pointers was born.

   			-- Dave Presotto

---

From: Mark Crispin <[email-masked]>
Date: Mon, 4 Aug 2008 19:39:20 -0700

+1

> back in the swamp you're trying to crawl out of.

This also includes removing the CAPABILITY command (not to mention the ENABLE command).

-- Mark --

PS: I told you so back in 1993.

_________________________________________________________________
Get Windows Live and get whatever you need, wherever you are.  Start here.
http://www.windowslive.com/default.html?ocid=TXT_TAGLM_WL_Home_082008
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Mon, 4 Aug 2008 19:46:54 -0700

> From: [email-masked]

I demonstrated opening and display a threaded scrolling view of 55,000 messages on a mobile device (specifically, both a Nokia N800 and an iPod Touch).  This was with Wi-Fi, but I demonstrated that the wall clock time wasn't substantially different using EDGE (via Bluetooth on the N800).  Furthermore, this was done with no cache of any messages on the mobile device.

So much for the claim for mobile devices can not do such a feat.

So much for the claim that mobile devices can not work without a cache.

This can be accomplished with IMAP4.

The question is whether or not it can be accomplished with IMAP5.

_________________________________________________________________
Reveal your inner athlete and share it with friends on Windows Live.
http://revealyourinnerathlete.windowslive.com?locale=en-us&ocid=TXT_TAGLM_WLYIA_whichathlete_us
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dave Cridland <[email-masked]>
Date: Mon, 18 Aug 2008 00:44:20 +0100

Back from holidays.

> for performance over slow links.
Features, not protocol syntax. At least, in my case. So yes,  
server-side sorting, threading, searching. MULTIAPPEND is protocol  
syntax, although I'd agree that some ability to pipeline appends is  
useful.

> searching, and sorting/threading.
No argument from me. (I'll answer Cyrus's argument about ENVELOPE  
distinctly.)

> That will happen.  Don't think for one minute that it won't.
You're probably right. I wonder if there's a way of making  
BODYSTRUCTURE simpler for clients without introducing silly states.  
Perhaps we simply need to work harder on showing how to use it well.

> Oh, you didn't mean that massive...  ;-)
I'd have to check FIND. But my inclination is toward something like I  
described in my point 3, so we end up with:

C: L01 LIST BASE "INBOX"
S: L01 LIST (\Attribute) "INBOX"
S: L01 OK "Completed"

C: L02 LIST SUB "INBOX"
S: L02 LIST (\Attribute) "INBOX/Foo"
S: L02 OK "Completed"

Note I've killed off the variable delimiter, all wildcarding, but  
I've been kind enough to keep to a strict path-like concept.

Searching for mailboxes is, of course, not met by this simplistic  
LIST - we could add additional, LISTEXTesque, criteria matching to  
handle this, but I find it hard to care - I don't think existing  
clients do this much.

> to identify which message to set \Answered when resuming a draft  
I think stable, fairly predictable, numeric references are good, I'm  
not keen on any other form. (I've read Chris's post on MSN/UIDs with  
interest, but I think removing UIDs entirely is a step too far -  
maybe I simply need to think on this more).

This means that PINE/Alpine would need to reverse the tactic, and use  
(what is now) UID SEARCH to translate MSN to UID, and therein use  
those. There's no loss to functionality.

On the other hand, this frees servers to send any event at any time,  
which masively reduces the complexity of the specification.

Of course, if you want real flexibility:

C: F01 FETCH (SEARCH 1:10 FROM crispin) BLAH
S: F01 SEARCH ALL 3:5,9
S: * 3 FETCH BLAH blah
[...]
S: F01 OK "Completed"

(As I read further along in my post-holiday backlog, I see you  
suggested something similar yourself later, so maybe this isn't as  
crazy as I thought).

> Remember too that POP3 uses MSNs.
IMAP is, of course, not POP3, and should not be limited so.

> days, only has not been called that.  It's the "O" status flag in  
Right, I'll accept that it adds no complexity in some cases. However,  
it certainly adds complexity in others, and it's rarely used, and the  
moment you start doing exciting things like having daemon agents  
processing your mail, \Recent is either worthless or breaks things.  
And that's merely covering the client-side impact - \Recent has a  
huge impact on the properties of the underlying store, which makes me  

> Ignore this warning at your peril.
I'm not ignoring it, but I would point out that having some common  
semantics is basically a good thing, and the wider these semantics  
are, the better. You rightly say that if the semantics are *too*  
wide, then they'll break things.

The idea here is not to define an IMAP5 store, but to expose a common  
model which fits existing stores. I think a lot of the interop  
problems of IMAP4 exist due to the comparitively loose model the  
store needs to exhibit, leading to a considerable amount of variance,  
and from there, a considerable amount of potentially untested  
behaviour.

> Do you intend to have responses carrying data?
I don't see why we should prohibit this now. I've had plenty of  
conversations with Alexey and others along the lines of "Oh, we can  
put this in a response code, that would be cleaner", "Oh, no, because  
we might need to use a literal/quoted string with ] in".

I don't want to have to think hard about the syntax, I just want to  
use it.

> If not, I fail to see the point.
Even with this one change, it makes handling response codes and  
terminal responses much easier, although that might just be personal  
opinion - but I do certainly prefer the more consistent syntax.

FWIW, I've always disliked the BODY[<token soup>] construct; I just  
don't have a better suggestion, quite yet.

> data model.  Do not destroy what you do not understand.
I'm not, I'm proposing this based on several years experience using  
this exact syntax with ACAP. :-)

> obvious examples), introduce data tags.  Do not overload command  
Please note I did *not* say "Remove untagged responses", which would  
indeed be throwing the baby out with the bathwater.

I do indeed think that the basic model of IMAP is right:

1) Data is send by the server to the client in principle at any time,  
but:
2) Servers will send the data if a command or an event causes them to  
do so.

The trouble is, I think IMAP also lacks the capability to send data  
relating specifically to a command, and that's a useful property.

Dave.

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 19 Aug 2008 08:52:14 -0700

On Sun, Aug 17, 2008 at 4:44 PM, Dave Cridland <[email-masked]> wrote:

>>  Features, not protocol syntax. At least, in my case. So yes, server-side
> sorting, threading, searching. MULTIAPPEND is protocol syntax, although I'd
> agree that some ability to pipeline appends is useful.

MULTIAPPEND is not just protocol syntax.  It requires that the server
process all of the adds atomically, and since it is unbounded (think
gigabytes of appends), it can be quite expensive to implement (though the
server can always say "NO" to something that it deems too expensive, of
course, this leads to user confusion quite quickly).

-Rob

-- 
Rob Siemborski | PGP:0x5CE32FCC | [email-masked]
Software Engineer | | 650-623-6925
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Aug 2008 09:25:42 -0700

> From: [email-masked]
> On Tue Aug 19 17:07:08 2008, Rob Siemborski wrote:
> > I believe the desired use case (as its been explained to me in the  
> > past) is
> > "I want to transfer a mailbox in its entirety, and I don't want to  
> > deal with cleaning up from partial failure."
> That could well be the original design criteria - is it what people  
> genuinely want or need now? Do c-client based clients use it with  
> this intent?

Yes.  When moving mailboxes from one server to another, complete
success is the only acceptable outcome.  Partial success is worse
than failure in this case.

The cloud providers get around the issue by requiring pull-moves and
being completely broken when doing push-moves (try doing a push
move to either gmail or @live to see what I mean).  A pull-move
requires giving authentication credentials to the destination.

It is fascinating how Lemonade was so obsessed with not giving
such credentials to submit servers -- hence URLAUTH -- that are
generally part of the same security web as the IMAP server, yet now
the same people are breathtaking in the sort of security information
they will give to third parties.

-- Mark --

_________________________________________________________________
Talk to your Yahoo! Friends via Windows Live Messenger.  Find out how.
http://www.windowslive.com/explore/messenger?ocid=TXT_TAGLM_WL_messenger_yahoo_082008

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 19 Aug 2008 09:07:08 -0700

On Tue, Aug 19, 2008 at 9:03 AM, Dave Cridland <[email-masked]> wrote:

> Right, I can go for the phrase "IMAP5 needs a method for atomic addition of
> multiple messages to the store". I'd reject the notion that this must be
> identical to MULTIAPPEND in syntax.
>

Sure.

Mind you, I'm not sure whether atomicity is required or not - I'm not sure
> whether it's efficiency or atomicity people want, basically.
>

I believe the desired use case (as its been explained to me in the past) is
"I want to transfer a mailbox in its entirety, and I don't want to deal with
cleaning up from partial failure."

-Rob

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Aug 2008 12:12:03 -0700

> From: [email-masked]
> On Tue Aug 19 17:25:42 2008, Mark Crispin wrote:
> Yes, I certainly see that for this case, you want atomicity - what
> I'm curious about is whether this is the case behind most usage of
> MULTIAPPEND today.

Given that I wrote the specification and the code that uses it, I ought to
have a clue as to how it is used.

The two cases are:
 1) push-transfer of an entire mailbox from one server to another.
 2) [al]pine copy of messages from one mailbox to another where the
destination is on a different server.

In both cases, atomicity is desired and necessary.

> I'm not sure I follow this - yes, Lemonade takes great pains to avoid
> passing plaintext equivalents across the wire, but I don't see the
> same people changing that?

Sigh.  Let me try again:

With push-transfer from server A to server B, the agent doing the
transfer knows the credentials for both server A (or may even be
server A) and server B, but does not have to disclose the credentials
of one to the other.

With pull-transfer from server A to server B, server B must be given
the credentials of server A.  This is exactly how gmail works; you
have to give it a login on a POP server so Google can suck your mail
down.  @Live and the other cloud providers do the same thing.

This is exactly the same security issue of forward without download.
Yet, in forward without download, it was likely that the two servers
would be in the same security realm (in the 80-90% case they will
be).  In this case, however, server B is likely to be in a completely
unrelated security realm -- yet you are giving it the keys to the realm
for server A.

Don't kid yourself; this is EXACTLY what the cloud providers want.
The entire purpose is to put themselves inside your security realm,
so that you can not readily remove them.  It isn't just MULTIAPPEND,
of course; there is a large laundry list of facilities which are marked
for death because they offer escape paths.  MULTIAPPEND is just a
tiny piece of the puzzle.

-- Mark --

_________________________________________________________________
Get thousands of games on your PC, your mobile phone, and the web with Windows®.
http://clk.atdmt.com/MRT/go/108588800/direct/01/
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Tony Finch <[email-masked]>
Date: Tue, 19 Aug 2008 21:55:08 +0100

> In both cases, atomicity is desired and necessary.

An alternative would be to make it possible for a client to synchronize
two (subsets of) mailboxes in a way that can recover from failures.

The MULTIAPPEND use-cases are related to the problem solved by Cyrus's
replication protocol, which can't be just IMAP because the replication
client needs to control UIDs (etc.) on the target server.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: VARIABLE 3 OR 4, BECOMING NORTHERLY 5 OR 6 IN NORTHWEST
LATER. SLIGHT OR MODERATE. OCCASIONAL RAIN OR DRIZZLE. GOOD, OCCASIONALLY
MODERATE OR POOR.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Aug 2008 20:45:09 +0100

> In both cases, atomicity is desired and necessary.
Yes, I see that's how you're using it - I was curious as to whether  
your usage is the only one.

I'd also say "desired", not always "necessary" - unless I'm really  
missing something.

> unrelated security realm -- yet you are giving it the keys to the  
It's this last bit I'm having trouble with - I'm not suggesting any  
such thing. I'm merely trying to figure out how clients are using  
MULTIAPPEND now, and whether those use cases support atomicity or not.

I don't think lack of atomic multiple appends precludes copying the  
messages between servers, it just makes it a little more tricky in  
the case of a failure. I've got the phrase "optimizing for the  
failure case" ready to type, but perhaps I'm missing something.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Aug 2008 11:46:19 +0200

> I'd also say "desired", not always "necessary" - unless I'm really  

Not everyone would agree. Some people think that if all messages can't 
be copied, copying most is better than copying none. I have heard 
explanations such as "the rest was all spam anyway". (Spam fails more 
often than legitimate mail because some servers dislike e.g. null 
bytes, which occur mostly in spam.)

> messages between servers, it just vmakes it a little more tricky in  

I don't think you're missing anything. If multiappend were really 
beneficial, tools like imapsync would support it.

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 20 Aug 2008 02:56:36 -0700 (PDT)

> Some people think that if all messages can't be
> copied, copying most is better than copying none.

Fuck it, that's fortune material ...

--lyndon

   It is much easier to suggest solutions when you know nothing about the
   problem.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Aug 2008 07:46:17 -0700

> On Tue Aug 19 21:55:08 2008, Tony Finch wrote:
> Which could be solved by transactions (eek!), or, potentially, by
> using some mechanism that says "Don't continue if you've emitted an
> error since point X".

And this is supposed to be a simplification of IMAP?!?

_________________________________________________________________
Be the filmmaker you always wanted to be—learn how to burn a DVD with Windows®.
http://clk.atdmt.com/MRT/go/108588797/direct/01/
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Aug 2008 07:51:16 -0700

> beneficial, tools like imapsync would support it.

mailutil supports MULTIAPPEND, and is used extensively for the
purpose of moving mailboxes.

Your argument is nonsense.  It could be used to say that if IMAP
was really beneficial, some POP3-only tool would use it.

---

From: Tony Finch <[email-masked]>
Date: Wed, 20 Aug 2008 16:01:06 +0100

On Wed, 20 Aug 2008, Dave Cridland wrote:
> Which could be solved by transactions (eek!), or, potentially, by using some
> mechanism that says "Don't continue if you've emitted an error since point X".

I wasn't thinking of anything as complicated as that. What I vaguely had
in mind was enough protocol features to allow the client to roll forward
or back in the event of a failure. Unfortunately this is somewhat stymied
by IMAP's model of an ordered append-only mailbox.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL MALIN: NORTHERLY 4 OR 5, BUT SOUTHWESTERLY IN SOUTH AT FIRST, AND
INCREASING 6 AT TIMES LATER. SLIGHT OR MODERATE. RAIN OR SHOWERS. MODERATE OR
GOOD.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Aug 2008 17:27:41 +0200

Mark Crispin writes:
> mailutil supports MULTIAPPEND, and is used extensively for the purpose 
> of moving mailboxes.

It has 20,000 google hits, imapsync has 68,000.

> Your argument is nonsense. It could be used to say that if IMAP was 
> really beneficial, some POP3-only tool would use it.

So it could, if that POP3 tool were the best-known open source tool for 
its purpose.

Popular open source tools attract patches and grow knobs.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Aug 2008 08:42:30 -0700

> It has 20,000 google hits, imapsync has 68,000.

I fail to see how this supports your conclusion that MULTIAPPEND is not
beneficial.

POP3 has 23,800,000 hits.  IMAP4 has 4,260,000 hits.

Perhaps the author of imapsync didn't know about MULTIAPPEND.  Or
perhaps he was waiting for other IMAP servers to adopt it.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Aug 2008 09:26:33 -0700

So why the hell aren't you asking me these questions?

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

> Subject: Re: [imap5] In response to the Crispinogram.
> Date: Wed, 20 Aug 2008 16:55:15 +0100
> From: [email-masked]
> To: [email-masked]; [email-masked]; [email-masked]
> CC: [email-masked]; [email-masked]
>
> On Wed Aug 20 16:42:30 2008, Mark Crispin wrote:
> Let's just ask him, it'll cut the speculation.
>
> Gilles - would you say that atomicity is a critical thing for IMAP?
> Would you like MULTIAPEND semantics in the base spec of a revision?
>
> And just as important, would you like to join this mailing list?
> mailto:[email-masked]?subject=subscribe
>
> Dave.
> --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Aug 2008 20:35:11 +0200

Mark Crispin writes:
> I fail to see how this supports your conclusion that MULTIAPPEND is 
> not beneficial.

I am presuming that if multiappend were important to a significant 
number of people, imapsync would get a patch. The credits file is long 
(and mixed).

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Aug 2008 11:51:37 -0700

> From: [email-masked]

That is a wretchedly bad argument, and it will come back to bite you.

_________________________________________________________________
See what people are saying about Windows Live.  Check out featured posts.
http://www.windowslive.com/connect?ocid=TXT_TAGLM_WL_connect2_082008
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Aug 2008 17:43:29 +0100

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Aug 2008 17:03:53 +0100

On Tue Aug 19 16:52:14 2008, Rob Siemborski wrote:
> server-side

Right, I can go for the phrase "IMAP5 needs a method for atomic  
addition of multiple messages to the store". I'd reject the notion  
that this must be identical to MULTIAPPEND in syntax.

Mind you, I'm not sure whether atomicity is required or not - I'm not  
sure whether it's efficiency or atomicity people want, basically.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 20 Aug 2008 17:45:09 +0100

> So why the hell aren't you asking me these questions?

Oh, I'd kind of assumed you were arguing that these things should be  
in, so I presumed to know your answers.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 20 Aug 2008 16:55:15 +0100

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Aug 2008 17:11:11 +0100

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 20 Aug 2008 14:48:14 +0100

Otherwise, if a series of pipelined appends fails once in the middle,  
the remainder would go through, and cause clients much more pain,  
since to "insert" a message you need to kill off the subsequent ones  
and so on. But to be fair, that's still not impossible, and if the  
offending message is deemed, by the user, to be unimportant, then  
continuing is fine.

> client needs to control UIDs (etc.) on the target server.

Right, that's a very different use case.

Dave.

---

From: <[email-masked]>
Date: Thu, 21 Aug 2008 22:42:47 +0300

What's the purpose of the imap5-request mailing list / the difference from the imap5 list?
Thanks. 

Best regards: Zoltán Ördögh
E-mail: zoltan dot ordogh at nokia dot com
Phone: +358 50 386 0566

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of ext Dave Cridland
Sent: 20 August, 2008 18:55
To: Mark Crispin; [email-masked]; [email-masked]
Cc: Discussion on drastically slimming-down IMAP.
Subject: Re: [imap5] In response to the Crispinogram.

Dave.
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade _______________________________________________

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 21 Aug 2008 11:06:23 -0700

Naive question:

   What I've heard as the task is to transfer an entire file from one venue to 
another.  Since tools already exist for transferring entire files, why should 
IMAP(5) also take on this task?

   I thought the goal of an IMAP effort at synchronization is to be active only 
with respect to *changes*, rather than simple, wholesale transfer.

   Besides that, I would assume that wholesale transfer of a file is a rare 
occurrence.  Do we really want to burden an interactive protocol like IMAP with 
a requirement to perform extremely rare functions?

d/

-- 

   Dave Crocker

---

From: Philip Guenther <[email-masked]>
Date: Thu, 21 Aug 2008 15:12:52 -0600

On Thu, 21 Aug 2008, Dave CROCKER wrote:
> Naive question:
> 
>   What I've heard as the task is to transfer an entire file from one 
> venue to another.  Since tools already exist for transferring entire 
> files, why should IMAP(5) also take on this task?

Right, and DNS should transfer zone data using some other protocol to 
transfer the file instead of kludging AXFR into the base protocol.  Heck, 
there's even a standard file format for DNS zone data that dates back to 
the original specifications and has demonstrated extensibility and broad 
support.  Why does AXFR exist and why did the DNS people continue to patch 
it (IXFR) instead of simply switching to rsync?

(Too bad there isn't a standard file format that encodes mailbox data and 
metadata completely and is broadly supported.)

Philip Guenther

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Aug 2008 14:19:42 -0700

Not a file.  A mailbox.

Most mailboxes are not files any more.  And file or otherwise, it is not
necessarily the case that the two systems in question use the same
data store format.

Note that the IMAP5 work is likely to prohibit the traditional UNIX format
of mail, at least without substantial non-standard extensions.  Ditto mh.

-- Mark --

http://panda.com/mrc

> Date: Thu, 21 Aug 2008 11:06:23 -0700
> From: [email-masked]
> To: [email-masked]
> CC: [email-masked]

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 21 Aug 2008 14:41:12 -0700

> transfer the file instead of kludging AXFR into the base protocol.  Heck, 

Are you suggesting that the justification for having the feature in IMAP5 is 
that some other protocol also incorporated the feature?  (I took your response 
and serious.  Forgive me if you didn't mean it that way.)

> metadata completely and is broadly supported.)

Creating a standard mailbox interchange format sounds like a nicely orthogonal 
effort...

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 21 Aug 2008 15:02:43 -0700

Mark Crispin wrote:
>> From: [email-masked] Mark Crispin writes:
>> I am presuming that if multiappend were important to a significant number
>> of people, imapsync would get a patch. The credits file is long (and
>> mixed).
>

I think this exchange highlights the challenge we have in finding an acceptable
basis for including or excluding features.

My own view is that web hits and anecdotal (personal) references and intuitions
are not likely to be very accurate or even very helpful in getting the group to
converge on a set of desired features and characteristics.

One thing I've noticed about quite a bit of the discussion, so far, is that it
is in terms of IMAP protocol constructs.  Obviously some of that detail is
necessary, but I suspect it is the wrong granularity for initial focus.

Examples of statements that strike me as being at the right level (but without
commenting on whether I personally agree or disagree with the specifics) are:

-----
> I wonder if mobile devices are included in the mass market definition?
-----
> I want to transfer a mailbox in its entirety, and I don't want to deal with 
> cleaning up from partial failure.
-----
> IMAP5 needs a method for atomic addition of multiple messages to the store
------
> I think stable, fairly predictable, numeric references are good, I'm not keen
> on any other form.
------
> Suppose we view IMAP5 as three feature packages:
>  IMAP5compat  (make it easy for IMAP4 clients to upgrade)
>  IMAP5core    (make it easy for basic mail clients to do basic operations)
>  IMAP5sync    (features for managing a local cache, lemonade stuff)
-----

These examples cover different levels of abstraction, but none is stated as IMAP
protocol details.  Instead, they are in terms of some claim of higher-level
utility or purpose.

Chris' "Here's a first strawman at a priority-ordered" note was a really good 
attempt to get discussion to target end-user utility, rather than immediately 
bog down in protocol detail.  So it's a shame it got no responses.

Really, folks.  If the list insists on debating protocol details, at this stage, 
the effort is going to be useless, because each debate about detail will lack 
any larger framework within which tradeoffs can be considered.

I also hope folk debate with an appreciation of the implication in the To: field
Display Name string for the mailing list that I copied:  "Discussion on
drastically slimming-down IMAP".  To drastically reduce is to leave out.  An
argument that something must be retained needs to deal with the fact that it is
already available in IMAP4.  If it's already there, then anyone wanting it can
use IMAP4.

What problems is IMAP5 trying to solve that IMAP4 does not?  (An answer like 
"IMAP4 is too complex" isn't helpful.)

What are the basic technical constraints on IMAP5 that distinguish it from IMAP4?

What higher level functionality might a user want to do that IMAP5 will *not*
support?

In order for the group to make progress, I suggest that the group's decisions
about what to include needs to have really massive/broad support, rather than
merely having one or two vigorous supporters and that no one person have
influence that is effectively equivalent to a veto.

d/
-- 

   Dave Crocker

---

From: Philip Guenther <[email-masked]>
Date: Thu, 21 Aug 2008 17:43:21 -0600

On Thu, 21 Aug 2008, Dave CROCKER wrote:
> > transfer the file instead of kludging AXFR into the base protocol.
> 
> Are you suggesting that the justification for having the feature in 
> IMAP5 is that some other protocol also incorporated the feature?

Not quite.

Right now, IMAP plus MULTIAPPEND solves the problem of "inject mailbox 
atomicly".  It sounds like you're proposing to solve that by instead 
defining a standard mailbox file format.  Implementations would then be 
expected to provide an "inject mailbox file" capability, apparently 
outside the protocol.  I'm not sure where you visualize the "generate 
mailbox file" action taking place.

I have a hard time seeing why implementations would bother with this.  
For a client-side implementor it's almost worse, as it's trapped between 
the existing possibilities:
A) IMAP without MULTIAPPEND gives inject w/o atomicity but 100%
   portability
B) IMAP with MULTIAPPEND gives inject w/atomicity

If someone wants 100% portability, they'll just grit their teeth and use 
(A).  If atomicity is important, (B) has better portability than the new 
format Right Now.  Is the proposal really simpler for implementation than 
(B) such that it can be expected to gain greater portability?

In addition, there is an existing mailbox file format that this would have 
to compete with: UNIX v7 mbox format (i.e., "From " start line).  It's 
crappy in various ways and underspecified, but it's entrenched enough to 
cut-off the low-end of the 'mental market' for mailbox file formats.

So let's look another protocol to see what its experience has been.  DNS 
already has a standard file format, so that hurdle to using a file 
transfer setup has never existed.  Despite that, the development of 
efficient file transfer and synchronization algorithms has not cut off 
work on in-protocol transfer.  It seems that at least in the DNS server 
market, the demand for supporting those extra-protocol transfer methods 
has not resulted in broad support.

Note that for DNS, the zone file format is useful in its own right, for 
specification and discussion.  Would a mailbox file format have similar 
side-benefits?  For DNS, the discussion is often at the level of just a 
few records and not a complete zone file.  Similarly, many email 
discussions involve just one message at a time without metadata concerns; 
RFC 2822 is generally sufficient for those.

So:
1) the lack file format may not be the major barrier (DNS comparison)
2) the format would have a niche application beyond what RFC 2822 gives
3) the overall "mailboxes transfer method" proposal seems to be in a bad
   position on the benefit-vs-deployability graph when compared to 
   existing solutions

> (I took your response and serious.  Forgive me if you didn't mean it 
> that way.)

Thank for you doing so.

Philip Guenther

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Aug 2008 17:52:51 -0700

> From: [email-masked]
> Mark Crispin wrote:
> I'm joining the discussion late here, so perhaps I've missed something
> that's already in the archives. Can you elaborate on this?

A great deal of the complexity in IMAP4 is to support legacy applications,
mailbox formats, and schemas.  It seems to be the sense of the majority
of the group that this is not desirable; that instead IMAP5 should define a
single ideal means of access and not give a damn as to whether or not it
can be implemented with an existing mail store infrastructure.

I disagree with that idea, attitude, and conclusion.  But I seem to be in the
minority.

I have my own ideas as to how a great deal of the complexity of IMAP can
be simplified.  I've already defined a framework in which the entire formal
syntax of IMAP can be on a single page, and now I'm working on some of
the commands based upon lessons of the past 20 years.  However, I also
believe -- and this is borne up by long sad experience -- that the best way
to kill a protocol is to have a committee design it.

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Aug 2008 04:41:11 +0300

> I disagree with that idea, attitude, and conclusion.  But I seem to be in the

I want to be able to use legacy mailbox formats - mbox in particular. I
don't really see anything discussed that would prevent me from doing
that. But yes, it could require more work from the server and it could
require changes to the directory layout (e.g. nothing prevents dual-use
mailboxes with mbox format).

> the commands based upon lessons of the past 20 years.

I don't think it's only the syntax that's the problem. The server
replies should be more strictly defined and the server design exposed
less to the clients. For example LIST replies can be hugely different
between servers even when the same commands are used to create the
mailboxes. I don't think clients should be expected to deal with that.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Dan Mosedale <[email-masked]>
Date: Thu, 21 Aug 2008 14:40:26 -0700

Dan

---

From: Tony Finch <[email-masked]>
Date: 23 Aug 2008 10:48:50 +0100

> files, why should IMAP(5) also take on this task?

The server needs to maintain consistency of its various indexes, beyond
just the contents of the mailbox. It probably doesn't represent the
mailbox as just a file, so it needs to translate the incoming data. If
you use a general-purpose file transfer protocol, how do the client
and server agree that this is a mailbox transfer that requires special
handling? Also, most file transfer protocols don't have the same security
model as IMAP and SMTP (SASL + TLS).

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 26 Aug 2008 10:56:55 -0700

ony Finch wrote:
> just the contents of the mailbox.

Maintaining application-specific semantics is always a good reason for a feature
that otherwise looks as if it is performed elsewhere, but actually isn't...

d/

-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 26 Aug 2008 10:56:30 -0700

d/

-- 

   Dave Crocker