Subject: Re: [imap5] Where is IMAP5 ?

From: Giovanni Panozzo <[email-masked]>
Date: Wed, 20 Jul 2011 11:32:54 +0200

Il 19/07/2011 14:07, Filip Navara ha scritto:
> On Tue, Jul 19, 2011 at 1:17 PM, Giovanni Panozzo<[email-masked]>  wrote:
>>
> All of these already exist.
> Calendar and contact synchronization is handled by CalDAV and CardDAV,
> which are much better suited for the job. They support time-range
> queries, free/busy lookup, scheduling and all related features.

I know: I'm using CalDAV to sync my brother's work calendar on his iPad
with the one on Gmail. It works great to sync calendar.
But I have no enterprise implementation of CalDAV, because the majority
of my customers has or want Exchange + Blackberry Enterprise Server.

My opinion is that having a single protocol/connection to access email,
contacts and calendar will be a big benefit:
- for client developers: they will avoid to write extra code to support
a second protocol. Less connections to open, less connecion to monitor,
less configuration dialog boxes to write to ask server address. Less SSL
certificates to manage. Less protocols to sniff when debugging.
- for server developers: IMAP servers already have good storage
functionalities and good commands to access them. Why calendar and
contacts related function (like search) can't be integrated in the code
of cyrus of dovecot ?
- for sysadmins: less configuration to do both at the client and the
server side.

> Autoconfiguration via DNS/HTTP exists as well. The DNS-based approach
> is described at http://tools.ietf.org/html/rfc6186. A mixed DNS/HTTP
> protocol is specified in Microsoft Exchange Autodiscover and it is
> compatible with IMAP and SMTP.

Can't see CalDAV or CardDav in RFC6186 :)
The basic idea behind Microsoft Exchange autodiscover.xml is good. I
mean "have the user only enter its email address, its password, and he
will be able to access all the PIM information he needs: e-mail,
calendar, contacts and whatever IMAP-nextgen will provide".
But autodiscover.xml is complex and was born to solve M$ Exchange
complexities. I would like to have something more IMAP related and a bit
simpler.

>> But the user, the sysadmin, the programmer, the market, they need and they
>> want these features, now.
>
> You're wrong on this one. The user wants fast and reliable email
> client (whatever that means), but they don't give a sh*t about the
> underlying technology. As long as user is concerned, BURL+GENURLAUTH
> is just as good as UID SUBMIT.

You are right, the user only wants fast and reliable mail.

What I really meant is: now configuring an IMAP+SMTP client is very hard
(read: impossible) for the average user.
Take an user with its brand new smartphone (my brother again, he is a
dentist, not an IT man). He taps randomly on the phone until he reach
the e-mail client program. If the client program only asks "just give me
your email address and your passord", your client and your protocol will
gain a new user.

If your software still ask the user (my brother) "tell me your e-mail
addres, your password, your imap server (uh ? what is it ?), whether it
supports PLAIN login or CRAM-MD5, whether the Sent folder is called
"Sent", "Sent items" or "Posta inviata" (the italian name), the user
will shut your software down and never open it again. And... i forgot to
mention namespace, folder separator, SSL selfisgned certs, SMTP server,
caldav server, carddav server and related options.

So in my opinion, the needs are
- One autoconfiguration file or DNS SRV records, mandatory in the next
version of IMAP.
- User message submission (UID SUBMIT) must be part of IMAP. And maybe
we should understand that user message submission (in IMAP) is different
from server message submission (SMTP). Servers and user have different
needs, so they deserve different protocols. Yes, I know, it will be hard
because of DSN or other cross-protocol problems. But it's not impossible.
To Gilles: my 3G bandwidth is still expensive. Small user error, like
attaching a 70MB video to an email when abroad, can cost ten of euros or
simply consume all montly traffic, leaving the user disconnected forever
during its travel.
(example: open K9 on android and attach a big video: your IMAP server
will APPEND it correctly, then your smtp server will refuse it... you
wasted traffic, a lot !).
And, for those that says that bandwidth doubles every 6 months, I can
say: yes, bandwidth raises, but also video files size raises and the
price per byte of 3G drops VERY slowly compared to them.

> Some programmers may prefer UID SUBMIT, but its too litte, too late.
> Even if you covered 99% of the use-cases you'd still have to convince
> the programmers to implement it and test it. Implementing a generic
> email client is already hard as it is. Adding another extension that
> may be supported by few servers is usually not worthwhile. The ones
> who care already implemented BURL+GENURLAUTH despite the fact that it
> is clumsy.

Yes, too little and is very late.
I can see only one possible solution: if IMAP5 will ever come to life,
it will need a working implementation immedialtey available. This means:
at least two mainstream servers and clients should implement it. We
can't do it on proprietary software like M$ Exchange or Outlook, but
maybe we can do it in cyrus, dovecot, K9, thunderbird...
And to simplify the implementation of IMAP5 in these products,
developers life must be as simpler as possible. This is why I don't see
CalDAV, CardDAV as a good solution. Yes, they can be source of
inspiration and their functionalities must be in IMAP5, but they are not
a fast path to the market. And it's late !

>> Maybe a "fork" of IMAP protocol outside IETF will do the trick ?
>
> Forking a dying project never solves anything. Many people tried to
> fork or redo IMAP over the years and it never gained adoption.

There are successful forks and unsuccesful ones. Like in every other
human project.
But... don't be afraid: I have no experience, time or resources to fork
such a big beast. I was just provoking :)

Giovanni

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 20 Jul 2011 12:54:05 +0300

> contacts related function (like search) can't be integrated in the code

Calendar functionality is very different from mail functionality. To
implement it efficiently it couldn't just be bolted onto existing mail
store, but rather use a whole new database design. So adding calendaring
to IMAP would not help server developers. For me it would be easier (and
cleaner and prettier) to build a whole new calendar server/binary than
to plug it into the existing IMAP code.

But I have to admit, I don't really like HTTP. I would have tried
implementing CalDAV already if it wasn't for its dependency on HTTP..

---

From: Filip Navara <[email-masked]>
Date: Wed, 20 Jul 2011 11:55:51 +0200

On Wed, Jul 20, 2011 at 11:32 AM, Giovanni Panozzo <[email-masked]> wrote:
[snip]
> of my customers has or want Exchange + Blackberry Enterprise Server.

There are enterprise CalDAV servers - Apple Calendar Server, Oracle
Beehive, Oracle Unified Communications Suite and several others.

> contacts and calendar will be a big benefit:

If we were to start from scratch then yes. We don't.

> certificates to manage. Less protocols to sniff when debugging.

Fine, HTTP is a good candidate for that. There's plenty of tools to
debug HTTP, atom feeds, WebDAV.

There were proposals to implement email protocols on top of HTTP. Not one, many:
- Hotmail used a WebDAV based email protocol.
- There was draft of Atom-based email access
(http://tools.ietf.org/html/draft-dusseault-httpmail-00)
- And there was at least one draft of WebDAV-based email protocol that
I can't find a reference to.

> contacts related function (like search) can't be integrated in the code

The requirements for email and calendars are VERY different. Just read
the CalDAV protocol documentation and the draft of CalDAV Scheduling
and perhaps you will understand.

Honestly, IMAP is just as good storage for calendars as FTP.

> Can't see CalDAV or CardDav in RFC6186 :)

http://tools.ietf.org/html/draft-daboo-srv-caldav-10

> complexities. I would like to have something more IMAP related and a bit

[snip]
> caldav server, carddav server and related options.

We don't ask any of that as long as the servers are configured correctly.

In the best case the user enters email address and password. Then we
determine the POP3/IMAP/SMTP/CalDAV/CardDAV/XMPP settings using
automatic discovery (Exchange Autodiscover, SRV records or fallback
.xml file distributed with the installation). We determine the folder
names using IMAP XLIST or SPECIAL-USE extension and folder layout
using the IMAP NAMESPACE extension.

Best regards,
Filip Navara

---

From: Filip Navara <[email-masked]>
Date: Wed, 20 Jul 2011 11:59:54 +0200

> complexities. I would like to have something more IMAP related and a bit

It's complex only if you make it complex. I've implemented it twice in
PHP on the server side and once in C# on the client side.

Best regards,
Filip Navara

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 20 Jul 2011 09:41:05 -0400

Hi Timo,

> to plug it into the existing IMAP code.

Very true. Some key differences include the fact that CalDAV does a lot of 
writing to the server (APPENDs) except that it needs to overwrite (modify) 
existing data - something that is very alien to IMAP (but could easily be 
"hacked" in). Then there is scheduling. The implicit scheduling model we 
have for CalDAV effectively means the server is sending messages internally 
between user accounts and nowadays, more often than not, externally to 
other calendar servers in the same cluster or across domains. Its likely 
that the approach to scaling a calendar server is also going to be very 
different from scaling IMAP. There is a lot of complexity there - but that 
is true no matter which "transport" protocol you choose to use.

> implementing CalDAV already if it wasn't for its dependency on HTTP..

Its just HTTP (and WebDAV) - there were several working server and client 
implementations at the very first CalConnect interop event many years ago - 
getting up to speed on a simple implementation is easy - doing scheduling 
right and scaling the whole thing is much harder. But again, just like 
IMAP, there are those who will just do the bare minimum to get by and those 
who try and do a thorough job of it.

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 20 Jul 2011 09:48:59 -0400

Hi Giovanni,

> - for sysadmins: less configuration to do both at the client and the

As noted in my reply to Filip, the way data is accessed and used in mail 
and calendar servers (well IMAP and CalDAV now) is very different so its 
not a simple thing to just bolt one set of functionality into the other 
(and yes there have been may people who have suggested to me that there 
should be a MailDAV protocol - and no I do not want to go there).

If we are going to forge ahead here, I would rather see a brand new 
protocol that can be data type agnostic (i.e. there is a lot more "stuff" 
than just email, calendar and contact data that a user might want to store 
and manipulate and relate - bookmarks, photos, music etc).

> Can't see CalDAV or CardDav in RFC6186 :)

See <https://datatracker.ietf.org/doc/draft-daboo-srv-caldav/>. That has 
been approved for publication but stuck in the queue on a missing reference 
for a very long time. I intend to bug ADs about this next week...

>>> But the user, the sysadmin, the programmer, the market, they need and
>>> they want these features, now.
>>

This is what RFC 6186 (and srv-caldav) is meant to solve. A user only need 
enter their email address and password and the client ought to be able to 
discover and configure all the relevant accounts via DNS SRV.

> - One autoconfiguration file or DNS SRV records, mandatory in the next

If we have an update to CalDAV I would also make DNS SRV mandatory ti 
implement there. We had a lot of issues with account discovery and setup 
earlier on which lead to the srv-caldav . Now that clients and servers are 
implementing that, things are better.

-- 
Cyrus Daboo

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 20 Jul 2011 17:02:32 +0300

On 20.7.2011, at 16.48, Cyrus Daboo wrote:

> If we are going to forge ahead here, I would rather see a brand new protocol that can be data type agnostic (i.e. there is a lot more "stuff" than just email, calendar and contact data that a user might want to store and manipulate and relate - bookmarks, photos, music etc).

Something like that could be interesting. I've always wondered why I couldn't simply have different kinds of views to all of my data. For example I might want to show my calendar entries as a list similar to emails. Or maybe showing emails in a view similar to a calendar. Things like that. Instead of having an "email client", I could have an "email view" and tell it to show either mails, local files, calendar, contacts, whatever.

A type agnostic protocol could be simply a database where you could specify what kind of types some specific fields were (dates, strings, numbers) and the server could figure out ways to optimize searching for those. Uh. Is this getting too close to SQL? :) Or was Notes something like this..

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 20 Jul 2011 19:49:29 +0200

> >>is just as good as UID SUBMIT.

Absolutely - so long as it works reliably, the user really doesn't care.

> >(read: impossible) for the average user.

And that's well before you start adding LDAP, CalDAV, etc into the mix.
Let along ACAP.

> This is what RFC 6186 (and srv-caldav) is meant to solve. A user
> only need enter their email address and password and the client
> ought to be able to discover and configure all the relevant accounts
> via DNS SRV.

Yes.  RFC 6186 is fricking awesome (as the kids used to say at one stage,
they've probably moved on without me).  It solves the problem that ACAP
really doesn't, which is reducing the amount of crap the user has to
deal with.  Mozilla did something with skanky XML over HTTP autoconfig
stuff for Thunderbird as well (I was semi-involved in that one).  Not a
great fan of the implementation choices, but I am a great fan of the
end result, which is that the user types in their username and password
and gets the correct settings.

> If we have an update to CalDAV I would also make DNS SRV mandatory
> ti implement there. We had a lot of issues with account discovery
> and setup earlier on which lead to the srv-caldav . Now that clients
> and servers are implementing that, things are better.

That would be great.

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 20 Jul 2011 19:57:57 +0200

> Something like that could be interesting. I've always wondered why I couldn't simply have different kinds of views to all of my data. For example I might want to show my calendar entries as a list…

I'm wary of building something "generic" to solve all the future's
problems rather than building a small thing that solves the current
problem well.  Get too generic and you don't produce benefits worth the
effort to implement.  We have a lot of protocols like that already - so
complex that nobody does them well.  Build a PHP by bodging together a
bunch of libraries in a useful way and make it trivial to get started,
and - much as it's butt ugly - you'll take over the world and make a lot
of people's lives better.

(did I say butt ugly, full of security problems, yes... but then it took
 over from the IMHO better mod_perl because it was easy for server
 administrators to install and sandbox, it was cross platform, and it was
 easy to just upload some .php files and start hacking with it.  Those
 features turned out to be more important than being "good")

> A type agnostic protocol could be simply a database where you could specify what kind of types some specific fields were (dates, strings, numbers) and the server could figure out ways to optimize s…

There are some good things about Notes, at the low level.  Unfortunately
the email component was crap last time I had to deal with it, both the
emailing bits and the user interface bits.

Bron.

---

From: Tony Finch <[email-masked]>
Date: Wed, 20 Jul 2011 19:20:59 +0100

Bron Gondwana <[email-masked]> wrote:
>
> Mozilla did something with skanky XML over HTTP autoconfig stuff for
> Thunderbird as well (I was semi-involved in that one).

Surely it can't be as badly designed as Microsoft's version.
http://fanf.livejournal.com/95018.html

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Malin, Hebrides: North or northwest 4 or 5. Moderate. Showers. Good.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 20 Jul 2011 22:38:20 +0100

On Wed Jul 20 15:02:32 2011, Timo Sirainen wrote:
> A type agnostic protocol could be simply a database where you could  
> specify what kind of types some specific fields were (dates,  
> strings, numbers) and the server could figure out ways to optimize  
> searching for those. Uh. Is this getting too close to SQL? :) Or  
> was Notes something like this..

ACAP - a schemaless protocol that handles objects with attributes you  
can slice and dice using client-specified comparators.

I tried implementing a calendar store in ACAP once, for a laugh.  
Turns out that you need server smarts too, and in spades, for  
calendaring, though. Mail I've not tried, but since each message is  
itself a tree of Things, I'm not sure it'd be great anyway.

I think, in general, that this is quite usual - you benefit  
enormously from the server knowing what data you're handling, in most  
cases. ACAP is very handy for the other stuff. (Perhaps that's a  
reason why it never took off - it's a lot of work writing a generic  
data server that then handles only un-sexy data)

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Giovanni Panozzo <[email-masked]>
Date: Thu, 21 Jul 2011 10:00:06 +0200

> I'm wary of building something "generic" to solve all the future's

Me too. Too "generic" usually means a complex protocol to handle higher 
level generalizations, and few lazy programmers won't be able to 
implement all the protocol specs.

But having IMAP5 handle only a limited number of objects (e-mail, 
contact/contact group, calendar/calendar entry, message filter) with 
their associated methods is not a bad idea, in my opinion.
The protocol specs will also benefit to have a direct 1:1 correspondence 
with an opensource Object Oriented API for clients and lazy programmers.