Subject: [imap5] My personal wishlist for an IMAP5

From: Philip Van Hoof <[email-masked]>
Date: Tue, 26 Aug 2008 18:37:02 +0200

I'll give it a try ...

I want to point out that for client developers IMAP4's biggest problem
is not the raw complexity of the protocol.

The problem with IMAP4 is this: 

	"You have to implement your requirements anyway."

Even if the developer of the IMAP server decided not to support a
specific capability. 

It doesn't matter for your requirements: you still need to implement it
in your client.

That explains why a lot of E-mail clients have horror code: clients need
to do both too. Your user doesn't care about how poor his own IMAP
server is.

What I want most from IMAP5 is a marketing word that says: THIS set of
modern capabilities IS an IMAP5 server. Once adopted E-mail clients
could far more easily consume the modern things:

It's a guarantee for quality for the user of your E-mail client: it's
IMAP5, so it's okay.

Right now it's like this: "It's an IMAP4, it might be okay, it might be
not much better than a POP server, we don't know yet. Let's try, shall
we?!"

Not good enough for an answer. You leave your user in doubt and he'll
blame it on "the device", "the computer", "the e-mail client", etc.

CONDSTORE:
----------

Remove and require QRESYNC's syntax in SELECT. No need for redundancy.
QRESYNC and CONDSTORE serve the same purpose. 

Newly written clients use QRESYNC, period.

SEARCH:
-------

Replace it with ESEARCH. Make ESEARCH a requirement for IMAP5

Make it possible to search multiple mailboxes (and find a solution for
the UIDs if you do, perhaps always send the mailbox name as a token in
front of the UID).

THREAD:
-------

Require it in IMAP5

LIST: 
-----

 * Get rid of that silly delimiter stuff. Who came up with that anyway??

 * Make LIST-EXT mandatory

 * Add counts

 * Make it possible to fetch n levels of hierarchy in the tree (I know
   about %, but I also want to fetch a few hierarchies with just one
   command)

NAMESPACE:
----------

Kill it after torturing it

LSUB:
-----

Kill it after torturing it twice

Folder removals
---------------

Add clarity about subscription state after removal of a folder without
first unsubscribing it.

Counts in LIST:
---------------

Every E-mail client needs to show an up-to-date count of each folder.

I know a lot of IMAP server developers say that they can't make LIST
fast if they need to add counts. But guess what E-mail client developers
have to do "anyway"? If they don't have the counts, they will do STATUS
on each folder. 

Yes they will.

It's in their UI requirements. 

Yes the user wants these counts instantly.

Due to having to call a bunch of STATUSes it's noticeable by the user
that the counts are not immediate. Not good. Makes IMAP look sluggish.

The server has an ENTIRE DAY to count. My E-mail client must satisfy the
user NOW.

Unsolicited events
------------------

The problem with unsolicited events in IMAP4 is that they are
underspecified. As a result it seems that each IMAP server behaves
different.

The net result of that is that most E-mail client developers simply
ignore most unsolicited events. Except the ones that can't be ignored
(EXPUNGE for example).

You can't rely on something that is not sufficiently specified and of
which the behaviour differs per IMAP server.

NOTIFY and IDLE are heading in the right direction (they specify the
events far more detailed, so it makes IMAP more reliable for the client
developer).

Unsolicited LIST events must also include the counts of the folders

Unsolicited EXPUNGE & XGWMOVE
-----------------------------

Kill it. It makes tracking sequence numbers required. Replace it with
QRESYNC's VANISHED: Less complex, more reliable. 

Add the mailbox name to VANISHED too, this makes it possible to tell us
about expunges in mailboxes that are not selected.

SELECT:
-------

Require QRESYNC in IMAP5

Make SELECT-ing folders less necessary (multi-span searches, VANISHED
with the mailbox name, etc).

STATUS
------

Kill it if LIST does it already

UNSEEN in STATUS (in LIST)
--------------------------

The number is useless. What the user perceives as "unseen" equals to:

	-> Not deleted and not read

Not read and yet deleted means "spam" (you don't count that as unseen in
the user interface). If not for ESEARCH there's currently no way to get
the right unseen count.

Most E-mail clients are counting the unseen after synchronizing (after
fetching all the flags of all the messages) for that reason.

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Rob Siemborski <[email-masked]>
Date: Tue, 26 Aug 2008 09:47:42 -0700

> UNSEEN in STATUS (in LIST)

For that matter, the concept of "flag as deleted" is so in conflict with how
most users and clients treat email, that IMAP5 should instead implement some
form of trash.  If this is done by "find flagged deleted messages across all
folders efficiently" that'd be fine by me.

-Rob

-- 
Rob Siemborski | PGP:0x5CE32FCC | [email-masked]
Software Engineer | | 650-623-6925
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Philip Van Hoof <[email-masked]>
Date: Tue, 26 Aug 2008 19:02:40 +0200

On Tue, 2008-08-26 at 09:47 -0700, Rob Siemborski wrote:

[cut]

> For that matter, the concept of "flag as deleted" is so in conflict
> with how most users and clients treat email, that IMAP5 should instead
> implement some form of trash.  If this is done by "find flagged
> deleted messages across all folders efficiently" that'd be fine by me.

I never needed physical Trash mailboxes. Flag as deleted works fine. But
only if we (client developers) would have these things:

- Multimailbox search for \Deleted (I would indeed search all mailboxes
  for all E-mails that are \Deleted. And the IMAP server must indeed be
  fast at returning me that). This is what your proposed, so indeed.

- ESEARCH so that I can do interesting searches on all \Deleted

- LIST (or STATUS) with an UNSEEN that makes sense. Because the current
  UNSEEN just doesn't help at all.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 26 Aug 2008 10:28:53 -0700

> It's a guarantee for quality for the user of your E-mail client: it's

That is an utterly hopeless cause.

No matter what the protocol says is mandatory, you will have server
implementors who do whatever they damn well wish even if it violates
the protocol.

In some cases, such as Microsoft, it is a legitimate blunder.  You just
have to spend weary years getting them to take the problem seriously.

In other cases, such as Google, it is intentional.

During my recent job search, I was lectured that I had to get over this
quaint notion that interoperability is important or even desirable.

Because of this, I have become convinced that it is hopeless to do
anything in open standards beyond a general framework.  Complex
behaviors such as IMAP semantics are hopeless until such time as the
EU (or some other legal authority) imposes fines on vendors for
standards non-compliance.

Until that time, there is no protocol police, nor is there any guarantee
that IMAP4, IMAP5, or IMAP69 means anything other than some vague
resemblance to what the document purportedly says.

-- Mark --

_________________________________________________________________
Be the filmmaker you always wanted to be—learn how to burn a DVD with Windows®.
http://clk.atdmt.com/MRT/go/108588797/direct/01/
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 26 Aug 2008 20:38:28 +0300

> resemblance to what the document purportedly says.

I guess you could trademark the name and have a set of requirements  
before allowing others to advertise compliance. I've no idea what IETF  
thinks of such practises though. :)

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 26 Aug 2008 13:40:17 -0400

Hi Philip,

>   UNSEEN just doesn't help at all.

Nope, not good enough. A client is going to need to present a view of the 
Trash mailbox complete with message listings. If the messages are scattered 
about in different mailboxes, some of which may not be cached, the client 
would have to go select each one to fetch the message envelopes. That 
simply won't do. If we want to do trash properly (and I am not necessarily 
advocating that) then we have to have a proper "move to trash" delete 
action and there has to be a proper way to uniquely identify the trash 
mailbox.

-- 
Cyrus Daboo

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 26 Aug 2008 20:43:45 +0300

On Aug 26, 2008, at 8:40 PM, Cyrus Daboo wrote:

> --On August 26, 2008 7:02:40 PM +0200 Philip Van Hoof <[email-masked] 
> >
> wrote:
>

Or alternatively support for virtual mailboxes (or at least virtual  
Trash) that allow a client to select a virtual Trash mailbox  
containing all \Deleted messages in all mailboxes.
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Philip Van Hoof <[email-masked]>
Date: Tue, 26 Aug 2008 20:01:48 +0200

> thinks of such practises though. :)

I was just writing the same proposal! :-)

And specify a complete test suite. Not just write one.

If you pass it, you can call your implementation IMAP5. If you don't,
you can't.

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 26 Aug 2008 11:09:07 -0700

On Aug 26, 2008, at 10:38 AM, Timo Sirainen wrote:

> before allowing others to advertise compliance. I've no idea what  
> IETF thinks of such practises though. :)

The IETF can help you publish a Standards Track RFC that lists a bunch  
of protocol extensions defined as a profile or bundle, and the overall  
requirement is "If the server advertises support for BundleFoo, then  
it MUST implement all of this blah."   I don't know if trademarking a  
name does anything to help or hinder this.

The IETF is not the compliance police and I don't think the IETF has  
anything to do, ever, with saying some implementation is or is not  
compliant.  However, sometimes there are other organizations that do,  
to a greater or lesser extent.  Some organizations (CalConnect, SIPIT)  
just run no-fault interop tests events, and no negative results are  
ever publicized -- this maximizes participation of volunteers at the  
expense of being able to shame people.  Other organizations will run  
compliance testing in their own labs, and rate implementations -- this  
is not always voluntary on the part of the implementors in question,  
and it's expensive to do, and I don't really know what the business  
model is.

The first kind of volunteer participation interop tests can also be  
organized during IETF week if you coordinate with me and Chris over  
not impacting the WG meeting schedule.  The Apps area has few enough  
meetings lately that this can work.

Lisa

---

From: Mark Crispin <[email-masked]>
Date: Tue, 26 Aug 2008 11:59:05 -0700

> thinks of such practises though. :)

One can always dream.  When I first started specifying IMAP, the idea was
that social pressure would ensure compliance.  I was young and naive
back then, and failed to recognize that it didn't work back then either.

-- Mark --

_________________________________________________________________
See what people are saying about Windows Live.  Check out featured posts.
http://www.windowslive.com/connect?ocid=TXT_TAGLM_WL_connect2_082008
_______________________________________________
imap5 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/imap5

---

From: Mark Crispin <[email-masked]>
Date: Tue, 26 Aug 2008 12:11:55 -0700

> And specify a complete test suite. Not just write one.

Test suites are hard.  Damn hard.  Among other things, they are a moving target; and all too often are reactive to the problems of the past rather than to present or future problems.

Timo has done outstanding work in his test suites; but I doubt that he claims that they do much more than test certain aspects of multiple access.

> If you pass it, you can call your implementation IMAP5. If you don't,

 Who says that "you can't"?  Who enforces this?

Mark Crispin says that certain well-known servers are not compliant IMAP4rev1 servers.  That statement, plus $4 or so, gets Mark Crispin an espresso at Starbucks.

Two of the biggest companies in the world have servers that violate explicit statements in the IMAP4rev1 specification.  The open source world isn't much better.

I spent 22 years of my life trying to specify out a protocol that a rigid enough syntax that would enforce interoperability.  I find it to be highly amusing that people with much less experience think that they can do a better job.  Do so with my blessings; but don't be surprised if you end up creating more non-interoperability.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 26 Aug 2008 12:16:45 -0700

> compliant.

This is simultaneously the IETF's greatest strength and its greatest
weakness.

-- Mark --

---

From: SM <[email-masked]>
Date: Tue, 26 Aug 2008 13:32:08 -0700

At 12:11 26-08-2008, Mark Crispin wrote:
>Two of the biggest companies in the world have servers that violate 
>explicit statements in the IMAP4rev1 specification.  The open source 
>world isn't much better.

The IETF publishes the specifications.  It doesn't certify 
implementation conformance and it unfortunately cannot stop people 
from violating explicit statements.

>I spent 22 years of my life trying to specify out a protocol that a 
>rigid enough syntax that would enforce interoperability.  I find it 
>to be highly amusing that people with much less experience think 
>that they can do a better job.  Do so with my blessings; but don't 
>be surprised if you end up creating more non-interoperability.

That won't stop them from trying.  :-)

Regards,
-sm

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 27 Aug 2008 12:47:08 +1000

On Tue, 26 Aug 2008 12:11:55 -0700, "Mark Crispin" <[email-masked]> said:
> Test suites are hard.  Damn hard.  Among other things, they are a moving
> target; and all too often are reactive to the problems of the past rather
> than to present or future problems.

I for one wouldn't mind having all the problems of the past sorted to start
off with.

Speaking as a server developer (having done a bunch of stuff with DAV servers
recently-ish) having something like litmus around to test a bunch of edge-cases
was fantastic.  It meant I could cheaply and quickly test my implementation
(and I wasn't starting from scratch, I was pretty massively rewriting and
extending an existing server).

The best thing writing suite buys you - a real urge to simplify your
protocol enough that it's viable to write the tests for it.

And the reason why google isn't IMAP4 compatible is pretty obvious.  The
protocol has too much impedance mismatch with the way their mail is stored.
They implemented it because it's the only even semi-well-supported protocol
that allows leaving mail on the server while managing it in a local client.
That doesn't mean it is a good match.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 27 Aug 2008 06:12:16 +0300

> back then, and failed to recognize that it didn't work back then either.

But with trademarks you could sue anyone not in compliance, that's a bit
more pressure to get it right. Although even there it would probably
help if the company owning the trademark was one of the big ones..