Subject: [imap5] Where is IMAP5 ?

From: Giovanni Panozzo <[email-masked]>
Date: Mon, 18 Jul 2011 11:21:46 +0200

Hi, I just landed in this mailing list after searching news on IMAP 
protocol.

Is this ML dead ?
Is Microsoft Active-Sync taking over IMAP ?

I'm a system integrator as main job, and sometimes a developer, C, Java, 
PHP for hobby. I manage some IMAP servers, mainly based on 
linux+dovecot+postfix.
But I can see that IMAP is being replaced by the simpler to configure, 
more features rich Microsoft Active Sync.

I have read the IMAP whishlist of "Philip Van Hoof", posted back in 
2008. It's a programmer's point of view whishlist, I like it, but I 
would like to contribute with my end-user and system configurator point 
of viwe. I also can contribute with debugging and testing.

But I can't see any traffic in this ML since 2010 :(

Is there anyone ? :)

Giovanni

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 18 Jul 2011 13:57:07 +0200

> But I can't see any traffic in this ML since 2010 :(

I think it got declared "too hard" and pretty much died - nobody could
agree on the important set of things to include.  Little things like:

"If you move 10,000 messages to Archive.2011 then all other clients
  shouldn't need to download everything again"

"If you rename a folder then every other client shouldn't need to
  download everything again"

Which in my opinion are important got jumped on by Mark "you don't
understand IMAP" Crispin - who has very fixed ideas about which problems
IMAP should be solving, and the ability to cache full mailbox state
locally for offline work is not one of them!

And then there's the "don't send the whole message up the wire twice on
compose" problem, which has been solved badly in a few ways but nobody
seems to be willing to look at "extend the APPEND syntax to add: also
send via SMTP on successful append", which is bloody trivial compared to
extending the SMTP server to make an authenticated request back to the
IMAP server and fetch out the body of the message, which is so complex
that approximately nobody gets it right, and certainly nobody actually
makes it easy for a mom-and-pop toaster server to set up.

Not to mention "move messages" - the number of times people run into
"I was trying to move these big messages to an archive folder and got
told I was over quota".  That's because IMAP doesn't do that, you COPY
first and then STORE+EXPUNGE the original messages.  A move command
that did nothing more than that but skipped the quota lookup if they
were both within the quotaroot would be enough.  It still has the
rare failure mode where a server problem during the copy+delete would
leave the message in BOTH mailboxes, but that's no different than if
the client crashes or loses network part way through the process.  So
what.  At least it would give a nice user experience in the common
case, and no extra downsides.

But there doesn't appear to be any will to actually solve these problems
on this list any more, and my proposal of a standard way to fetch the
digest of a message (DIGEST=SHA1) got shot down, so I kind of gave up
and just do my own thing now... and try to get upstream Cyrus to follow
all the standards that it can at the same time.

I notice gmail do pretty much the same, they don't care too much about
standards, but have some IMAP extensions that let you see that a message
is the same original message after a rename.  Their folders are a bit
funky as well, but they do expose the stuff to allow you to use them
efficiently.  The problem is, you can fetch DIGEST.SHA1 from FastMail,
you can fetch X-GM-MSGID from Gmail, and you can't rely on ANYTHING
in the general case, because you might wind up talking to an old
Lotus Notes server, and all bets are off.

That's the biggest problem that an IMAP5 would be good at solving,
lift the baseline so that clients don't need to work around lack of
UIDPLUS, CONDSTORE, etc.  Meaning they can get consistent semantics
without having to keep fallback codepaths that guess about what happened.

Bron.

---

From: Randall Gellens <[email-masked]>
Date: Mon, 18 Jul 2011 09:37:58 -0700

>  makes it easy for a mom-and-pop toaster server to set up.

Extending IMAP to do SMTP is one of those "attractive nuisance" 
solutions that is seductive at first, but increasingly problematic 
when examined.  Particularly when combined with SMTP's own complexity 
and set of extensions, it quickly becomes worse than the alternatives.

>  without having to keep fallback codepaths that guess about what happened.

That was also one of the goals of lemonade -- move the IMAP baseline 
so clients didn't have to deal with every possible server variation. 
Problem is, it's hard for a client to refuse to work with older 
servers.  A lemonade client has an advantage in that it's presumably 
in a new device and a new environment, but it would be even harder 
for a client to be IMAP5 only.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
[Television is] chewing gum for the eyes.
    --attribributed to Fred Allen via Frank Lloyd Wright

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Jul 2011 18:09:34 +0100

> makes it easy for a mom-and-pop toaster server to set up.

Yes I agree that it's really nasty, but it does have some kind of
architectural logic.

A key principle is that an MTA should not have to look in the message data
to deliver a message - just the SMTP envelope (i,e, the MAIL and RCPT
commands). This leads to various ESMTP features (notably DSN) glomming
stuff on to the envelope. Many of these features include things that an
MUA would typically like to control.

This leads to the problem that an IMAP extension to support message
submission would have to duplicate much of ESMTP, including support for
as-yet-undefined ESMTP extensions. In particular you can't simply tweak
APPEND since it has no provision for specifying the message envelope.

I tend to think that the original principle is a historical relic that
should probably be discarded (though that would require re-doing the DSN
extension, and perhaps others, leading to all sorts of compatibility fun).
That would allow you to do message submission without an envelope,
including directly from a message store. But the argument has been done
and dusted...

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Humber: Southwest 5 to 7 decreasing 4. Moderate or rough, becoming slight or
moderate. Squally showers. Moderate or good.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 18 Jul 2011 20:34:29 +0200

On Mon, Jul 18, 2011 at 09:37:58AM -0700, Randall Gellens wrote:
> when examined.  Particularly when combined with SMTP's own
> complexity and set of extensions, it quickly becomes worse than the
> alternatives.

Pigs arse.  It's only complex if you make it complex.  Pipe the message
to an external command (say: postfix's 'deliver', or sendmail -bm) and
be done with it.  Not hard.  Sure you can make the problem overly complex
and then shoot down the strawman, and people have been for years.  But
a simple way of saying "store this email and also cause it to be delivered"
would solve 99% of people's problems adequately, without the current mess
of uploading approximately the same message twice, and without every email
client on the planet also having to be an SMTP client if it wants to run
on an operating system that doesn't have a "deliver" of its own.

Add a couple 

> for a client to be IMAP5 only.

Sure, for a while.  But you could create a new client that's only IMAP5
and say "this client only supports the shiny new IMAP5" and people would
say "we want IMAP5 on our server".  While you say "Lemonade" and people
say "huh?"  Lemonade isn't marketable the way IMAP5 is.

That said, I don't much care what it's called.   I care that insanely
baroque complex protocols are built to work around rare edge cases
(IMAP server doesn't have a working command that can take an email and
deliver the bloody thing somewhere) and over 10 years since I started
taking an interest in email programs, almost every client in existence
STILL pushes the email via SMTP and then appends it to the Sent folder,
because the alternatives are crap.

Seriously, "attractive nuisance" indeed.  I'm tempted to implement this
and write patches for a couple of clients.  It's probably about 50 lines
of code in Cyrus (including error checking) to make this work.  Set up
the append, invoke deliver with the path to the file on disk, and either
abort the append or complete depending on the result of the deliver command.
Add a few lines of config file to allow you to specify an alternative.  No,
scratch that - there's already an external deliver command used for sieve:

{ "sendmail", "/usr/lib/sendmail", STRING }
/* The pathname of the sendmail executable.  Sieve invokes sendmail
   for sending rejections, redirects and vacation responses. */

Bron.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 18 Jul 2011 20:45:02 +0200

> MUA would typically like to control.

Explain sendmail -bm.   Most people just want the email to get put into
an outbound queue thanks very much.  The key principle is nice in theory.
In practice your average MTA is doing plenty of deep content inspection
before it agrees to do anything with an email these days anyway.

> APPEND since it has no provision for specifying the message envelope.

APPEND (ACTION SEND ENVFROM "[email-masked]" ENVTO "[email-masked]") \
"Sent Folder" TIMESTAMP (FLAGS) {1234+}
....

99% solved.  The search for perfect has produced pretty-much "unusable".
Sure there are plenty of syntactical ways to skin this particular cat,
but a key value list as the second parameter to APPEND (or you could put
one after FLAGS too) would work fine.

> including directly from a message store. But the argument has been done

And still nobody has actually come up with anything which actually works.
Which sounds to me like "dusted under the carpet".

I'm not interested in solving all the world's theoretical problems, I'm
interested in solving the "this message needs to go into the user's sent
folder and also be sent to the receipients named in the headers" problem.
In the simplest possible way.  Simple enough that there's a high
probability of clients and servers actually implementing it, because
it costs them so little and gains them so much.

Bron.

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Jul 2011 19:58:45 +0100

Bron Gondwana <[email-masked]> wrote:

> Explain sendmail -bm.

The command line controls the envelope. See the -N and -B options for
examples of SMTP service extensions, the -f option gives the argument to
MAIL FROM, and the non-option arguments correspond to the RCPT TO
arguments.

> Most people just want the email to get put into an outbound queue thanks
> very much.  The key principle is nice in theory. In practice your
> average MTA is doing plenty of deep content inspection before it agrees
> to do anything with an email these days anyway.

Yes.

Tony.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 18 Jul 2011 21:00:21 +0200

> MAIL FROM, and the non-option arguments correspond to the RCPT TO

Sure, or you can just pass a message and it will read the headers to
determine the sender and recipients.  Sometimes you don't want that,
but that's a much rarer case that plenty of clients don't really support
anyway.  We're optimising the rare cases at the expense of making the
common case easy.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 11:47:10 -0700 (PDT)

> locally for offline work is not one of them!

Troll.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 18 Jul 2011 21:09:47 +0200

> >locally for offline work is not one of them!

Absolutely.  However your response to most of my arguments about the
things that are causing friction amongst users and frustrating their
wants is "you don't understand IMAP".

Fact: lots of people want the ability to cache pretty much "the entire
mailbox state", and not have to download it all again.  Example from
##imap earlier today:

<BranchPredictor> brong: having RENAME not change the mailbox UIDVALIDITY would be far cooler ;-) at least for me.
<brong> BranchPredictor, asked and answered I'm afraid
<BranchPredictor> yes, I remember.
<brong> you have to, or you can wind up with a different folder with the same uidvalidity and different contents
<BranchPredictor> anyway, thanks for help.
<brong> anyway, what do you win?
<brong> but not changing UIDVALIDITY on rename?
<brong> (sorry, by)
<brong> unless you're implementing "undo" for a mistaken rename
<BranchPredictor> not really. just don't like the idea of redownloading entire 84MB of data (for ~85000 messages). :)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

<BranchPredictor> I'm not insisting on anything, really, there are servers that don't have stuff like UNSELECT or UIDPLUS implemented and THAT is real PITA for me, because I still need to handle them somehow.
<brong> BranchPredictor, even with "RENAMEPLUS" (i.e. the client should know that it renamed the folder) it only helps the client that actually did the rename
<brong> to do anything else, there would have to be a "folder unique id" that the client could use to detect renames
<brong> Cyrus has one internally
<brong> but there's no way via IMAP to expose it
<BranchPredictor> well
<BranchPredictor> time for IMAP5? :-D

Lots of people don't like the idea of redownloading entire 84MB of
data on a folder rename.  They're crying out for a mail protocol
that offers them this.  IMAP is an "attractive nuisence" that has
stopped anyone except the closed crowd from developing anything
else, because it's kinda almost good enough, if you jump through
enough hoops, to do what they want to do.  And everyone supports
it, or at least near enough, or at least kinda some parts.

Which is why we still have clients downloading 84MB of data when
someone renames a folder.  Because it's rare enough that the
pain becomes a distant memory again next time they want to do it.
Or the just don't rename folders.

And NOBODY, and I mean NOBODY, enjoys configuring a separate SMTP
server at their ISP and doing the "will my provider support port
587, or will port 25 be blocked for me" dance to get email to send.
A few years ago it was particularly bad, when most clients defaulted
to trying port 25, and most ISPs were blocking port 25.  Most of our
support traffic was people suddenly discovering they couldn't send
email to our servers any more, and us talking them through using
alternative ports.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 12:10:54 -0700 (PDT)

On Mon, 18 Jul 2011, Bron Gondwana wrote:
> We're optimising the rare cases at the expense of making the
> common case easy.

bccs, address forwardings, and mailing lists are "rare cases"?

-- Mark --

http://panda.com/mrc

---

From: Randall Gellens <[email-masked]>
Date: Mon, 18 Jul 2011 12:02:10 -0700

Solutions which seem to work but don't handle anything except the 
most obvious case can be dangerous.  For a while it was common for 
people to use fetchmail to retrieve mail via POP and then relay it 
via SMTP.  This is one of those solutions that seems to work but 
because it only has access to the data (the envelope being gone) can 
result in unpleasant problems.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
The radio listener saw nothing; he had to use his imagination...With the
high cost of living and the many problems facing him in the modern
world, all the poor man had left was his imagination.  Television has
taken that away from him.
          --Fred Allen

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 12:14:44 -0700 (PDT)

> wants is "you don't understand IMAP".

That is the inevitable result of your refusal to understand, and your
false representation of the issue (and the positions of others) when you
talk with others.

-- Mark --

http://panda.com/mrc

---

From: Joshua Cranmer <[email-masked]>
Date: Mon, 18 Jul 2011 12:15:13 -0700

> bccs, address forwardings, and mailing lists are "rare cases"?

And is this anything that couldn't be resolved with
APPENDNSEND <list of recipients> <message> ?

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 18 Jul 2011 21:07:09 +0200

> "Sent Folder" TIMESTAMP (FLAGS) {1234+}

Write draft-gondwana-imap-send, make it cover 99%, submit it for 
publication without support for all the corner cases.

Please.

We need more ninetyninepercentery and less perfection.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 12:19:59 -0700 (PDT)

> APPENDNSEND <list of recipients> <message> ?

That would certainly solve it as long as you duplicate all of SMTP and its
states.

The problem is that you have to duplicate all of SMTP and its states.

-- Mark --

http://panda.com/mrc

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 18 Jul 2011 21:22:43 +0200

> result in unpleasant problems.

Sure.  Except if you can optimise the most obvious case you can
actually improve life for most users, most of the time.  I'm
not proposing anything radical like "discard SMTP submission".
I'm just saying that if all you want to do is store a message to
your sent folder and have it dropped into a deliver queue, then
making it too complex to implement will be worse than making it
simple.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 12:23:17 -0700 (PDT)

> publication without support for all the corner cases.

That way, we can point to the document whenever anyone in the future
brings up this bad idea...along with the fact that nobody implements it
and why.

I-Ds and RFCs are littered with such worthless documents.

-- Mark --

http://panda.com/mrc

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 19 Jul 2011 00:35:13 +0200

> The problem is that you have to duplicate all of SMTP and its states.

Hi Mark, I was wondering -- what is the use case that is not covered by
the sample command Joshua proposed? What feature of the SMTP protocol
should I look at in order to fully understand this?

I haven't read the ESMTP specification carefully, so I'm just very
vaguely familiar with a simple SMTP conversation, and not all of the
extensions, hence this honest question. What feature of a MUA depends on
these states?

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

---

From: Gilles LAMIRAL <[email-masked]>
Date: Tue, 19 Jul 2011 00:36:12 +0200

Hello,

> mailbox state", and not have to download it all again.  

It's easy, just get enough header allowing unique identification of any
message in a cache and then no matter folder names where messages are stored
you have the entire whole account/mailboxes state.

> <BranchPredictor> time for IMAP5? :-D

Time to think about better email clients.

> that offers them this.  

People could cry for better client email software thinkers too.

> Or the just don't rename folders.

Another way to solve this issue is
- have just one INBOX
- use IMAP flags keywords as a way to sort messages.

No suffer for renaming folders this way, just flags renames, IMAP4 
can do it easily and fast.

In fact I wonder why IMAP Gmail didn't implement that, it's the
first thing coming to mind. Instead the keyword-is-a-folder Gmail 
way is not economical with bandwidth using basic IMAP.

> 587, or will port 25 be blocked for me" dance to get email to send.

Do you want to send mail using your private mailbox hoping the postman 
will open it and know what job to do?

Don't you confuse Post Office issues with personal mailbox issues?

-- 
Au revoir,                             09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

---

From: Randall Gellens <[email-masked]>
Date: Mon, 18 Jul 2011 15:30:18 -0700

At 1:57 PM +0200 7/18/11, Bron Gondwana wrote:

>  standards

Someone made an interesting observation at a recent IETF: if email 
were being done today, we probably wouldn't have any standards. 
Instead, each service provider would publish their own APIs, and 
anyone wanting to do an interoperable client would implement each set.

-------------- Randomly selected tag: ---------------
Chance that an American cannot name a single right protected by the
First Amendment: 1 in 3.                           (Harper's Index)

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 19 Jul 2011 01:00:16 +0200

> anyone wanting to do an interoperable client would implement each

Yep.  That would kinda suck.  Then again, that's what we do anyway.
Maybe not at the low level, but if you aren't publishing a whole pile
of DNS records and adding signature headers and providing spam feedback
loops, then you're struggling to interoperate anyway.

I haven't been to any of the conferences and I haven't been as involved
in the standardisation world as I should have been.  Maybe I should
start doing that.

On the flip side to that argument, if email was being built by the 
process this list has taken with IMAP5, we'd all be working on something
like GNU/HURD, while upstart Linux just went ahead and solved people's
problems.  It's a balance and there are always groups pulling from both
sides.

I doubt gmail did XLIST because they wanted to write something new - they
did it because there was nobody solving the problem, and they needed it
solved now.  Not one day, when it had been bikeshedded to death in
committee.

And it was needed enough that others jumped on board.  Now it's got a
standard and the world is migrating towards the standard (SPECIAL-USE).
I have my doubts that SPECIAL-USE would have happened if Google hadn't
just gone ahead and started implementing.

Bron.

---

From: Curtis King <[email-masked]>
Date: Mon, 18 Jul 2011 16:15:43 -0700

> Time to think about better email clients.

I think there already are better email clients and they keep on improving thanks to mobile devices. These clients are from major vendors as well. I quick check of the extensions being used on my server, UIDPLUS LITERAL+ UNSELECT NAMESPACE ESEARCH IDLE QRESYNC COMPRESS=DEFLATE.

Looking at the protocol the clients are being very smart to reduce the amount of data being sent over the network.

ck

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 17:27:58 -0700 (PDT)

> People could cry for better client email software thinkers too.

This has been my lament for many years, and it has come to a head with a
miserable state of email clients for Android.  The default email client is
completely unusable; an alternative called K-9 Mail is semi-usable.
[Don't get me started on the reptile excrement called Mail.app on iOS.]

These programs betray both "cookbook" programming and the inability to
envision heavy-duty mail volume.  On the other hand, they do a great job
in not deleting messages even when you ask them to do so; perhaps this is
a law enforcement requirement.

Sadly, there is no market feasibility to make a mobile device client that
doesn't suck.  The development cost to do mobile device email right is
around US $1 million; and the market isn't there to recover the
development costs.

Nor are email clients the make/break app for a particular mobile device.
Oh, it was for RIM, years ago; but RIM got complacent and is heading down
the toilet.

> - use IMAP flags keywords as a way to sort messages.

This is, in effect, what Gmail does; but it doesn't expose it that way to
IMAP.

> way is not economical with bandwidth using basic IMAP.

The answer for why they did that is simple; doing it this way fools
Outlook into a user view that is somewhat like the Gmail view.  As far as
Google is concerned, Outlook is the only IMAP client that matters.  Never
mind that Outlook is first and foremost an Exchange client.

Gmail's behavior confuses the hell out of an IMAP client that follows the
specification; but that whole point is to confuse (in desirable ways) a
client that does not follow the specification.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Mon, 18 Jul 2011 17:54:02 -0700 (PDT)

> should I look at in order to fully understand this?

It's not one single feature; it's the entire infrastructure formed by
SMTP, its extensions, and its agents.

One important aspect, that isn't in the protocol at all, is message
injection and the policy related to it.

The moment you have another agent that can inject, you either have to
duplicate the policy (with the risks that go along with that) or make the
agent proxy to SMTP (with all the subtle interactions that occurs when a
proxy crosses protocol boundaries).  Pay close attention to how error
conditions percolate, especially on the proxy/protocol boundaries.

It's easy/tempting to say "it doesn't matter".  We in the commercial email
software world sometimes encourage this thinking among the free software
guys.  Our customers pay us large(!!) sums of money because we say that it
matters.

Good luck.

-- Mark --

http://panda.com/mrc

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Jul 2011 08:49:15 +0100

> it costs them so little and gains them so much.

As a data point, the whole BURL thing has been implemented multiple  
times now.

As someone who implemented it, I find it unexciting and slower in the  
general case (like sending this message), but very *very* handy in  
the case where I'm forwarding very large attachments.

Now, could this be all implemented as an APPEND (or similar)  
extension? Sure. You wouldn't want to do it as append, actually,  
you'd want a new SUBMIT command, for a whole bunch of reasons.

The core problem you'd run into is that you've introduced an  
architectural shift.

For every facility offered by the MHS relating to submission, you  
need to potentially do it twice.

The only solution I see is to make the IMAP[45] command such a thing  
wrapper around ESMTP/Submission that it offers no clear benefit,  
except for the ability to avoid BURL/GENURLAUTH. I think it'd be  
pretty ugly, but it *is* possible.

Now, you absolutely could just solve the minimal case, but that also  
has the effect of closing the door on all future development, so I'm  
less than convinced that's a workable idea, either.

The trick is *not* covering 100% of the existing submission cases;  
the trick is including sufficient extensibility to make 100% parity  
possible.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Jul 2011 09:16:31 +0100

On Tue Jul 19 00:15:43 2011, Curtis King wrote:
> I think there already are better email clients and they keep on  
> improving thanks to mobile devices. These clients are from major  
> vendors as well. I quick check of the extensions being used on my  
> server, UIDPLUS LITERAL+ UNSELECT NAMESPACE ESEARCH IDLE QRESYNC  
> COMPRESS=DEFLATE.

Absolutely - you're referring to Apple, I think, but there's also  
Nokia's sponsored work on the Qt Mail Foundation classes which  
implement the trio as well. It's interesting that Android hasn't  
caught up yet, but I think that's in part due to lower software  
reusability on that platform. This is what spelt the death-knell for  
Symbian, so it'll be interesting to see what happens there.

Of course, mere use of extensions isn't nearly as important as  
efficient use of IMAP, and - to drag us kicking and screaming back  
onto the topic - I think it'd be useful to examine why this hasn't  
happened before we decide that a SUBMIT command is the solution.

Dave.

---

From: Jan Kundrát <[email-masked]>
Date: Tue, 19 Jul 2011 10:22:01 +0200

> conditions percolate, especially on the proxy/protocol boundaries.

I have to confess I'm a bit lost here. I work as a Linux sysadmin, and
each server we work on usually has a properly configured
sendmail/postfix/whatever, if only for local submission. Each of these
MTAs provide a command-line facility for queuing a message for
submission (the traditional `sendmail -bm` interface), which puts the
message into the MTA's spool and arranges for the final delivery. It
works pretty well, provided your MTA is properly configured (and I guess
that that's a reasonable, if new, assumption for an IMAP server).

Therefore, it seems to me that simply adding a `UID DELIVER 123` command
(valid in a selected state, expecting an UID of a message to be
submited) would be very trivial, especially given that there's already a
Sieve command for doing exactly that, as someone already mentioned.

On the other hand, you and Dave both state that this is not going to
fly. I do respect your experience, and am not trying to avoid my
homework here, it's just that I simply don't see any reason why such a
simple scheme won't work. So, could you please clarify what am I missing
here?

With kind regards,
Jan

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Jul 2011 01:48:52 -0700 (PDT)

On Tue, 19 Jul 2011, Dave Cridland wrote:
> Absolutely - you're referring to Apple

Apple's iOS client is vulnerable to server differences in extensions, to
the point that it does not work reasonably with servers that implement
extensions in ways that it does not expect.

I am told that Apple will remove their use of at least one of these
extensions for this reason.

> happened before we decide that a SUBMIT command is the solution.

The general problem with extensions is that the interoperability support
matrix is much more complex.

This is a fairly widespread problem, and leads to the "it works with every
other client, therefore your client is broken" situation.

This even happens with the IMAP base specification [*], which has an
extremely stern compliance doctrine where you can unambiguously say "A is
right and B is wrong".  Most extensions lack this stern doctrine, and thus
have multiple possible compliant implementations that do not interact.

[*] Case in point: I have had to argue with a server vendor that
mis-implemented ENVELOPE and claimed that resulting client behavior was a
client bug and not a bug in the server.

-- Mark --

http://panda.com/mrc

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Jul 2011 10:24:57 +0100

> simple scheme won't work. So, could you please clarify what am I  

For the *simple* case, it'll fly:

UID SUBMIT 123

It's not hard to cope with BCC and other cases of needing an  
envelope, although that's not at first obviously needed, as you saw:

UID SUBMIT (FROM [email-masked]) (TO [email-masked]) 123

Unfortunately, that's not yet "done". There's considerations like  
DSNs, and so on.

Now one option which I think Bron suggested is that the SUBMIT  
command deliberately, by design, only handles the simple cases and no  
more. That leaves you safely in the field of hand-waving and passing  
the buck, but in practical terms it means that a client implementer  
wanting to implement advanced features still uses Submission, whereas  
implementers who don't care, don't.

The trouble is that produces a two-tier arrangement, and that's never  
been successful in the past - if it's successful, then the more  
complex path will die. You may cheer, but it means those features  
have to go somewhere, and they'll end up shoe-horned into the SUBMIT  
command, putting that back to square one.

Architecturally, there is a global MHS, which is ESMTP. Currently,  
there's no architectural break between submission and transmission,  
but a SUBMIT command injects such a break, and the model of the  
global MHS needs to be accessible through such a command. This is  
hard enough now, when we have to consider exposure of DSNs, etc, but  
it's even harder if we wanted to create, say, a recall function,  
since we now have to create protocols in two places, instead of one.

X.400 works this way, incidentally, and does so fine - but it was  
designed from the ground up to support this mode. An X.400 MUA can  
submit mail either via the MHS protocol (P1, I think) or via the  
store protocol (P7 - again, I think). But the models are kept very  
strongly aligned - I think it's even the same PDUs involved.

I have not said that any of this is not possible.

I *am* saying that a quick hack will have knock on issues, and  
ultimately these issues will need to be dealt with.

It's my opinion - and I think you'll see if you read closely that  
it's the same as that of Mark, Randy, and Tony - that the level of  
work required to do this probably exceeds the benefit.

Dave.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Jul 2011 02:50:37 -0700 (PDT)

Thank you Dave.  Inline comments below.

> For the *simple* case, it'll fly:

Nominally, yes.  However, this ignores the effect of Murphy's Law.

Experience of those of us who lived through the 1970s, 1980s, and early
1990s, when there were many mail programs that behaved that way, indicates
that Mr. Murphy enforces his law.

> Unfortunately, that's not yet "done". There's considerations like DSNs, and
> so on.

DSNs are a Very Big Concern for some people who pay a lot of money to have
them handles right.

> The trouble is that produces a two-tier arrangement, and that's never been
> successful in the past

Read: "has always been disastrous".

In cases where one tier is entrenched (and SMTP has 29 years of
entrenchment), the other tier doesn't stand a chance.

> It's my opinion - and I think you'll see if you read closely that it's the
> same as that of Mark, Randy, and Tony - that the level of work required to do
> this probably exceeds the benefit.

Whenever I encounter simplistic "solutions" to very complex problems, I am
reminded of Albert Ghiorso's review of the story of the "radioactive boy
scout", in which he concludes "[Hahn] seemed to think that he had a new
idea that no one had thought of, but that notion could only come about
through ignorance...In the end, knowledge is needed, and that comes about,
I have found, only through education, study, and lots of experience."

-- Mark --

http://panda.com/mrc

---

From: Giovanni Panozzo <[email-masked]>
Date: Tue, 19 Jul 2011 13:17:11 +0200

After reading 24 hours of posts, I'm rather disconcerted.

1 year to not decide for "UID SUBMIT" is too much. How long will it take 
to decide to include features like calendar and contact sync ? And 
autoconfiguration via DNS/HTTP ? Yes, they are all features outside the 
current IMAP scope, and this will lead only to strong opposition in this ML.

But the user, the sysadmin, the programmer, the market, they need and 
they want these features, now.

Unfortunately there is an opponent, ActiveSync, which includes all. And 
now it's the winning one.

IMAP is dying, I have read a lot of good ideas on this ML to let it 
survive. But I have read too much opposition too.

Maybe a "fork" of IMAP protocol outside IETF will do the trick ?

---

From: Filip Navara <[email-masked]>
Date: Tue, 19 Jul 2011 14:07:28 +0200

On Tue, Jul 19, 2011 at 1:17 PM, Giovanni Panozzo <[email-masked]> wrote:
> 1 year to not decide for "UID SUBMIT" is too much. How long will it take to
> decide to include features like calendar and contact sync ? And

All of these already exist.

Calendar and contact synchronization is handled by CalDAV and CardDAV,
which are much better suited for the job. They support time-range
queries, free/busy lookup, scheduling and all related features.

Autoconfiguration via DNS/HTTP exists as well. The DNS-based approach
is described at http://tools.ietf.org/html/rfc6186. A mixed DNS/HTTP
protocol is specified in Microsoft Exchange Autodiscover and it is
compatible with IMAP and SMTP.

We implement all of these in our product.

> But the user, the sysadmin, the programmer, the market, they need and they
> want these features, now.

You're wrong on this one. The user wants fast and reliable email
client (whatever that means), but they don't give a sh*t about the
underlying technology. As long as user is concerned, BURL+GENURLAUTH
is just as good as UID SUBMIT.

Some programmers may prefer UID SUBMIT, but its too litte, too late.
Even if you covered 99% of the use-cases you'd still have to convince
the programmers to implement it and test it. Implementing a generic
email client is already hard as it is. Adding another extension that
may be supported by few servers is usually not worthwhile. The ones
who care already implemented BURL+GENURLAUTH despite the fact that it
is clumsy.

Also, however good you make the specification there will always be
implementation quirks and we, as client program writers, have to deal
with these quirks. Many vendors don't care about bugs in their IMAP
implementation and we have to workaround them in various ways. It is
problematic to explain to the user that something is broken on the
server if it works with Microsoft Outlook or Mozilla Thunderbird.

> Unfortunately there is an opponent, ActiveSync, which includes all. And now
> it's the winning one.
>
> IMAP is dying, I have read a lot of good ideas on this ML to let it survive.
> But I have read too much opposition too.
>

Forking a dying project never solves anything. Many people tried to
fork or redo IMAP over the years and it never gained adoption.

Best regards,
Filip Navara

---

From: Joshua Cranmer <[email-masked]>
Date: Tue, 19 Jul 2011 10:05:11 -0700

On 7/19/2011 2:50 AM, Mark Crispin wrote:
>> DSNs, and
Hmm, I had thought that DSN was merely embedded into the message 
envelope, although closer inspection has informed that this thought was 
wrong.

I'll be honest that I don't care all that much about "only post the 
message once when sending" ...

---

From: Martin Konold <[email-masked]>
Date: Tue, 19 Jul 2011 08:29:35 +0200

On Montag, 18. Juli 2011 20:34:29 Bron Gondwana wrote:

Hi Bron,

I wish more people would be as pragmatic as you. (Yes I am disappointed where 
LEMONADE went. When I back then proposed an improved IMAP extension for IMAP5  
I was directed at LEMONADE instead)

> I'm tempted to implement this
> abort the append or complete depending on the result of the deliver
> command. Add a few lines of config file to allow you to specify an
> alternative.  No, scratch that - there's already an external deliver
> command used for sieve:

I am willing to also work on this on the server side (e.g. putting your cyrus 
patches in Kolab) and on the client side (initially KONSEC Konnektor for 
Outlook later maybe adding support to KDE Kontact/Kmail).

What about the following approach:

- Extend IMAP APPEND to allow also for delivery via SMTP.
- APPEND to a special mailbox "outbox". (The name of this special folder shall 
not really matter but its special role is controlled using a folder 
annotation)
- the IMAP server takes care about sending the mail (either synchronously(*) 
while the APPEND happens or asynchronously,) Lets call this process spooler.
- The spooler then moves the message from "outbox" to "sent-mail". Again the 
name of the folder is arbitrary but the role is controlled by a folder 
annotation.

(*) Synchronous mode seems attractive for immediate feedback to the user of 
the client. I think it is useful for simple checks like correct syntax of smtp 
addresses. On the other hand asynchronous error handling is required anyway 
due to the asynchronous nature of SMTP. 

-- 
--  
e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Sitz: Adolfstraße 23, 70469 Stuttgart, Partnerschaftsregister Stuttgart PR 126
http://www.erfrakon.com/

---

From: Martin Konold <[email-masked]>
Date: Tue, 19 Jul 2011 08:46:47 +0200

On Montag, 18. Juli 2011 21:23:17 Mark Crispin wrote:

Hi,

> I-Ds and RFCs are littered with such worthless documents.

Why are you shooting down contributions by experienced people actually doing 
real work and delivering working solutions to millions of people?

IMHO I would like to see such a draft and I am willing to provide feedback 
improvements and in case I am convinced implementations for clients and 
servers.

It is a good tradition to look at RFCs after the drafts have been implemented 
by at least two parties and tested in real world environments.

Yours,
-- martin

---

From: Randall Gellens <[email-masked]>
Date: Tue, 19 Jul 2011 11:58:05 -0700

At 1:17 PM +0200 7/19/11, Giovanni Panozzo wrote:

>  And autoconfiguration via DNS/HTTP ?

Does anyone remember ACAP?  Or IMSP?

-------------- Randomly selected tag: ---------------
Men and nations may sometimes act rationally when all other
possibilities have been exhausted.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Jul 2011 12:41:37 -0700 (PDT)

> now it's the winning one.

Have you done any real (as opposed to .NET cookbook) development in
ActiveSync?

To what extent do you use ActiveSync in a high-volume production
environment?

How do you address the problem that users don't see the complete message
content because ActiveSync's tranformations removed it?  Do you just
pretend [as Microsoft does with @Live] that the content never existed?

ActiveSync is in vogue because Exchange is in vogue; and will cease to be
in vogue at the advent of the Next New Thing that replaces Exchange.  Not
long ago, Novell was all the rage; but now that Groupwise is being
replaced by Exchange, few people care to spend much new development effort
on Novell protocols and cries of "death to eDirectory!" are commonly
heard.

Surely you don't think for one minute that Apple, Google, and goodness
knows who else implements ActiveSync because they like it?  In case you
haven't noticed, their ActiveSync client support is specifically to
support Exchange servers; their native technologies for messaging,
calendaring, contacts, etc. is not ActiveSync.

Lately, I've been collecting stories of departments paying good money
(some of which I have pocketed!) to establish their own open standards
(IMAP, etc.) based email servers, because central IT went to ActiveSync
and the result was far less usable and reliable.

Even funnier are the press releases from central IT assuring their clients
that they will presently iron out all the glitches in their ActiveSync
services and the result will be just as good as what they had 15 years
ago...

> IMAP is dying

Cretins have said since 1990 that "IMAP is dying".  It hasn't happened
yet.

> Maybe a "fork" of IMAP protocol outside IETF will do the trick ?

Be my guest.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Tue, 19 Jul 2011 13:15:40 -0700 (PDT)

> real work and delivering working solutions to millions of people?

HAR HAR HAR.

It is always enjoyable to laugh at the nonsensical ranting of a fool.

I particularly liked "experienced people".  That's ROTFL material.

If they were experienced, they would know that their idea, far from being
new and novel, is very old (at least 25 years in my memory), has been
discussed to death many times, and has even been tried multiple
times...with the result always being a trainwreck.

-- Mark --

http://panda.com/mrc

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 20 Jul 2011 00:43:29 +0200

> bccs, address forwardings, and mailing lists are "rare cases"?

When you store the message via IMAP, the Bcc: header is intact.
sendmail is perfectly capable of parsing it out and using it,
at least the one that ships with postfix.  Forwarding without
rewriting the To: header is a rare case and not really a
candidate for "store to IMAP sent folder".  Mailing lists are
fine, the 'To:' header will be the mailing list.
-- 
  Bron Gondwana
  [email-masked]

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 20 Jul 2011 00:47:59 +0200

> Don't you confuse Post Office issues with personal mailbox issues?

God yes.  I would love that to just work.  Indeed, some places do
that/have done that.  They have a little flag you flip up to let
the postie know to check the box, since otherwise they wouldn't if
they had nothing to deliver.

Luckily, computers aren't physical postmen, and so your analogy is
pointless and crap.  Things on computers don't have to mirror
exactly the physical counterparts after which they were named.

One of my co-workers has the poster "you wouldn't download a car"
parodying the anti-piracy message on DVDs.  The tag line is
"fuck you, I would if I could".  If I could use one configuration,
one username, one password, one connection to do anything that would
be great.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Tue, 19 Jul 2011 23:51:07 +0100

> Does anyone remember ACAP?  Or IMSP?

*waves*

---

From: Gilles LAMIRAL <[email-masked]>
Date: Wed, 20 Jul 2011 04:04:51 +0200

Hello,

> If I could use one configuration,

Easy: just use a well written webmail and forget everything else, the car
is there too if you have imagination, just go where postmen are already 
nice to you.

Breaking the current postmen/mailboxes model just to save bandwidth that 
is doubling every 6 months is intense thinking, congratulations!

How it was when bandwidth was 33.6 kbps? already a no problem since
the solution has been doubling bandwidth every 6 months.

The other issue is that IMAP is not alone, there is POP to change too.

50 IMAP current server softwares implementations to be changed and learn 
them how to post locally while every email client software know currently
how to post anywhere. How good it will be, you deserve a poster!

We should merge all protocols into only one that will solve 
every past and future needs, how about [email-masked]?

42 == https?

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 20 Jul 2011 08:07:38 +0200

> is there too if you have imagination, just go where postmen are already 

That's nice is everyone has a car, sure.

> is doubling every 6 months is intense thinking, congratulations!

Here in Norway, many "post offices" are actually a counter at a
grocery store.  This means that you can access the full range of
postal services until 11pm.  Coming from Australia where the
post offices kept bank hours, it's quite a cultural difference.
It makes sense - here's somewhere that's already open and that
has similar basic skills and money handling facilities
available, why not use it.

Sure in some ideologically pure world you wouldn't even have
supermarkets - you would drive to the farmer's gate to buy
milk, and then drive off to a butcher to get meat and a
greengrocer for vegetables.  Funnily enough, people like
supermarkets where they can go do most of their jobs in one
place.

> the solution has been doubling bandwidth every 6 months.

It was shit - it's still shit.  I'm really surprised that nobody
actually came up with a solution that wasn't so annoying to
implement that nobody did it.

> The other issue is that IMAP is not alone, there is POP to change too.

Whatever.  POP doesn't have any way to upload email, so it doesn't
matter anywhere near as much for POP, because you're already not
saving the email back to the server.  You probably need a different
solution there.

> how to post anywhere. How good it will be, you deserve a poster!

The saving will be purely bandwidth, and simplicity for those who
are building both ends and hence can rely on the IMAP software
having this facility.  The point is - you don't have to use it,
but there are benefits large enough to be worth using it.  You
can still use regular SMTP for delivery instead if you need anything
more complex, or you don't want to make an IMAP append at the same
time.

But - in a crazily common case where you want to store an email to
your sent folder and also send it, this can halve the bandwidth use
and delay to the user.  Sure they could use webmail instead, that's
just a layer of middleware.  You could also implement this as a
piece of middleware that wrapped IMAP and SMTP in some custom XML
protocol and sent that to the client.  Hmm, I think we've already
done that except we used JSON.  It kinda sucks too.

> every past and future needs, how about [email-masked]?

Er, OK.  Hi strawman.  Perhaps we should merge every single shop
into one giant Walmart.  Except we don't.  Specialist shops
still exist for when you have specialist needs - or if you just
have a few everyday basics to get and the supermarket stocks the
things you need, you go to the supermarket and buy the whole lot
in one go.  I'm arguing for the the IMAP store to stock a basic
delivery method.

Bron.

---

From: Filip Navara <[email-masked]>
Date: Wed, 20 Jul 2011 08:17:11 +0200

>> Does anyone remember ACAP?  Or IMSP?

+1

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jul 2011 15:12:26 +0200

> is doubling every 6 months is intense thinking, congratulations!

Bandwidth isn't doubling every six months, unless you constrain the 
comparison to an IMO unrealistic degree.

I had about as much bandwidth yesterday as I had in 1995. The change is 
that yesterday I read mail on the train, which I couldn't do at all in 
1995. Last year I read mail in rural India, albeit with as little 
bandwidth than I had in 1990-2. Amazing progress, but IMO the real 
change is that the network has grown.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jul 2011 15:27:11 +0200

I wrote:
> Amazing progress, but IMO the real

Oh. And that bandwidth availability now varies wildly. Many more people 
have network access, but how much varies. For example, my smartphone 
says it has HSPA right now, but sometimes it falls back all the way to 
GPRS. Even within city limits. Quite a difference.

Arnt

---

From: Gilles LAMIRAL <[email-masked]>
Date: Wed, 20 Jul 2011 16:06:28 +0200

Hello,

> Bandwidth isn't doubling every six months, unless you constrain the comparison to an IMO unrealistic degree.

Yes I invented the value. No problem when arguing to someone inventing a "99%" argument.

Estimating my personal Internet bandwidth rise is a factor of 1000 in 15 years.
15 * ln(2) / ln(1000) = 1.5
So Internet bandwidth is doubling every 18 months, same law for local network (1 Mbps -> 1 Gbps).
Funny it's the same value as the Moore law for CPU power.
Maybe same law for disk storage.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jul 2011 17:00:12 +0200

> Yes I invented the value. No problem when arguing to someone inventing a "99%" argument.

I've no problem with the period. I've a problem with the statement that 
bandwidth doubles in any period of time. It's too narrow. Too many 
people are getting new slow connections, even people who had faster ones.

You'll want an example of both. One. If you read the right/wrong news, 
you might have seen an announcement that Sony Ericsson won a contract to 
deploy GPRS in Bangladesh. That adds perhaps 5-10 million new users with 
50kbps effective bandwidth. Two. Go visit a friend who has both an 
iphone and a fast DSL connection. Ask him casually to look up something 
in wikipedia. If he pulls his iphone from his pocket instead of walking 
over to the computer, the DSL connection has been "replaced" by the 
iphone's 2G/3G connection.

Arnt

---

From: Gilles LAMIRAL <[email-masked]>
Date: Wed, 20 Jul 2011 19:09:06 +0200

Hello,

> I've no problem with the period. I've a problem with the statement that bandwidth doubles in any period of time. 

It does double in every place.

> It's too narrow. Too many people are getting new slow connections, even people who had faster ones.

Their new slow connections doubles every 18 month too.

Slow connexions don't change the fact that optionnaly doubling sent messages transfer
is peanuts when writing messages at 1 byte per second. The big issue is that users 
don't care about message size.  It's just because email client softwares don't represent size
on the interface (nobody knows what Kb Mb Gb means). So users send everything over
email, they send 700Mb cdroms, 4 Gb dvds, they do try to send that. They're on their 
right because the reality exposed by email client softwares just let them think it
is possible.

The issue is not sending a dvd twice, the issue is sending X dvds (twice or not)
while receiving 100*X dvds a day. This is a never ended pure bandwidth issue or, 
better, a size representing issue (this one can be solved). 
It's not an "IMAP must know how to post locally" issue.

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 21 Jul 2011 08:42:01 +0200

On Wed, 20 Jul 2011 15:03 -0700, "Randall Gellens" <[email-masked]> wrote:
> At 8:07 AM +0200 7/20/11, Bron Gondwana wrote:
> 
> >  POP doesn't have any way to upload email
> 
> There's XTND XMIT, which was a disaster.

There is only one benefit here, reusing the same connection. An IMAP based equivalent is that plus the bandwidth saving.

So tell me how it was a disaster precisely? Other than a lack of capability information meaning clients tried it when it wasn't supported?

The "considered harmful" arguments all sound remarkably like Hurd enthusiasts railing against the "dirty" design of Linux.  Mostly it's the lack of compelling benefits and the shithouse interface for configuring Eudora that I see looking for information.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Dave Cridland <[email-masked]>
Date: Thu, 21 Jul 2011 08:37:12 +0100

On Thu Jul 21 07:42:01 2011, Bron Gondwana wrote:
> There is only one benefit here, reusing the same connection. An  
> IMAP based equivalent is that plus the bandwidth saving.
> 
> 
And a fanatical devotion to the pope.

> The "considered harmful" arguments all sound remarkably like Hurd  
> enthusiasts railing against the "dirty" design of Linux.  Mostly  
> it's the lack of compelling benefits and the shithouse interface  
> for configuring Eudora that I see looking for information.

That's an impressively random lash-out.

I thought configuring Eudora from a standard ACAP server or  
WorldMail's cut-down one was really quite straightforward.

Dave.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 21 Jul 2011 17:29:08 +0200

> And a fanatical devotion to the pope.

Obviously, you can't get anywhere without having one of those,
or at least professing to carry one when in polite company.

> WorldMail's cut-down one was really quite straightforward.

Ok - last I looked was ~1998 when I last used Eudora admittedly.
Perhaps it got a whole lot better in the meantime.

It certainly wasn't as easy a the first screen offering an easily
selectable "use ACAP server for all configuration" and then entering
a server name, username and password.  Which is still approximately
one item too many - which is why "enter your email address and
password and the client configures itself correctly" is the direction
things are going to.

But it certainly seemed that nobody was actually doing it 10 years
ago, choosing instead to publish big piles of screenshots on how to
configure their systems with IMAP/POP.  I don't buy the theory that
it's because they were all morons who didn't understand ACAP.

Bron.

---

From: Randall Gellens <[email-masked]>
Date: Thu, 21 Jul 2011 16:39:40 -0700

At 8:42 AM +0200 7/21/11, Bron Gondwana wrote:

>  So tell me how it was a disaster precisely? Other than a lack of 
> capability information meaning clients tried it when it wasn't 
> supported?

Oh, that wasn't a problem.  There were more serious security issues 
(from lashing together different protocols to implementation issues 
with executing other programs) and a complete lack of support for 
SMTP extensions.

-------------- Randomly selected tag: ---------------
It is impossible to make anything foolproof because fools are so
ingenious.

---

From: Gilles LAMIRAL <[email-masked]>
Date: Fri, 22 Jul 2011 02:49:36 +0200

Hello Bron,

> available, why not use it.

Ok. It's available where you live so you can use it.
Don't ask all grocery stores of the world to become post offices,
it will simply increase difficulty to become a grocer.

> supermarkets where they can go do most of their jobs in one

Inside the supermarket you don't find meat at the same place
than fish and you don't ask fish to the butcher.

>> How it was when bandwidth was 33.6 kbps? 

> It was shit - it's still shit.  

Just for you. Nobody else cares about halfing bandwith on
1% of email traffic.

> I'm really surprised that nobody

Webmails, Exchange, Notes can implement that.

> saving the email back to the server.  You probably need a different

You say it right. One side is sending a message, another one is
saving a message. How IMAP could handle all errors than can 
occur during a email submission? By becoming a SMTP server.

In fact, many IMAP servers hosts are not SMTP server hosts that 
can send correctly any message from a local post, they just 
know how to deliver locally to the IMAP server. So your 
proposition requires more configuration and dependencies, not less.

> having this facility. 

No. How IMAP can submit locally? smtp port? pipe? unix socket? shared memory?
forking command? It will oblige IMAP sysadmins to configure IMAP server 
softwares so they can post locally. More work than configuring 
a SMTP server independently.

> The point is - you don't have to use it,

IMAP servers have to support it, and sysadmin too.

You really need it? Just hack any imap server software and
any email client you use and happiness will come.

> but there are benefits large enough to be worth using it.  

No. 
If benefits are large enough, code it for yourself.

> and delay to the user. 

Email sending and saving is done in the background so users never wait.
Half bandwidth on sent messages is peanuts.

> into one giant Walmart.  Except we don't. 

Internet from my screen is like a giant Walmart.

> I'm arguing for the IMAP store to stock a basic delivery method.

You arguing for the IMAP store to become a sub-MUA to be used by full-MUA.

---

From: Giovanni Panozzo <[email-masked]>
Date: Fri, 22 Jul 2011 11:01:54 +0200

> than fish and you don't ask fish to the butcher.

But we used the car, find a place to park, walk inside the supermarket 
one time (and not two). We saved a lot of time and money. Then, inside 
the supermarket we can talk to the butcher or take fish.

I hope the same will happen for IMAP5: one single portocol for access 
(write less code, configure less parameters), but access more services.

> Just for you. Nobody else cares about halfing bandwith on

I care. Sometimes that 1% traffic is the most expensive traffic you have 
paid for in the last 3 years. I happened to me last week with my 3G 
connection and my android.

> occur during a email submission? By becoming a SMTP server.

Wrong. With UID SUBMIT, IMAP servers will become also SMTP clients, not 
SMTP servers. We just move SMTP the client logic from the user client to 
the server.
The user can still have all the notifications he likes via the IMAP 
channel or via a returning DSN.

> a SMTP server independently.

A little configuration work for one person (the server sysadmin) will 
save a lot of work for the user and helpdesk people to configure SMTP on 
hundred devices. When we used IMAP+SMTP I lost a lot of time at the 
phone with the user just to discover that... the network he was using is 
blocking outgoing SMTP (port 25) and sometimes also blocking SMTPS.

Two channels (IMAP+SMTP) creates a lot of user support requests at your 
helpdesk service.
Three channels (IMAP+SMTP+*DAV) means more user support again, because 
http/https could also be blocked, transparent proxyed and many other 
horrible things.

A *LOT* of user support at the phone.

This is one of the points because my IT chiefs decided to move to 
Blackberries or ActiveSync in the last 5 years. End user configuration 
and support !

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 22 Jul 2011 11:19:42 +0200

> blocking outgoing SMTP (port 25) and sometimes also blocking SMTPS.

In addition to that, as someone in this thread already mentioned, if
your IMAP server supports server-side filtering via Sieve, the Sieve
implementation already has to support mail submission.

I believe that it can be assumed that if someone runs an IMAP server,
she also has access to a properly configured MTA for mail delivery.
Configuring, say, a Postfix instance to relay e-mail from the IMAP box
to the already existing MTA can be done by an average sysadmin in less
than five minutes. I'd be surprised if other platforms made this harder.

Cheers,
Jan

---

From: Filip Navara <[email-masked]>
Date: Fri, 22 Jul 2011 11:24:24 +0200

On Fri, Jul 22, 2011 at 11:19 AM, Jan Kundrát <[email-masked]> wrote:
[snip]
> she also has access to a properly configured MTA for mail delivery.

In most cases, yes.

In cases where the IMAP server is used for accessing archives then
there's no MTA usually. Similarly, Evernote used to have IMAP gateway
that allowed accessing notes stored on the server and there was no
MTA.

Best regards,
Filip Navara

---

From: Jan Kundrát <[email-masked]>
Date: Fri, 22 Jul 2011 11:31:02 +0200

> that allowed accessing notes stored on the server and there was no

One could argue that IMAP servers fulfilling similar roles should simply
return a NO to the DELIVER/SEND/SUBMIT/SENDMAIL/... command to indicate
to the user that message submission has to take place elsewhere. Or not
advertise that extension at all, if it's an extension and not a core
feature.

Cheers,
Jan

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 22 Jul 2011 12:59:00 +0300

> advertise that extension at all, if it's an extension and not a core

Such servers would fail APPEND anyway, so they couldn't even get that
far..

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 22 Jul 2011 12:07:13 +0200

> that allowed accessing notes stored on the server and there was no

So don't advertise the SUBMIT or DELIVER or whatever you want to call
it capability and the client will know it can't send email through
that connection.  Not rocket science.

Bron.

---

From: Filip Navara <[email-masked]>
Date: Fri, 22 Jul 2011 12:16:51 +0200

> that connection.  Not rocket science.

Sure, I just wanted to remind people that there are other use cases.

I remember that someone was proposing SUBMIT to be mandatory and
that's definitely not a good idea.

Best regards,
Filip Navara

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 22 Jul 2011 12:40:48 +0200

> that's definitely not a good idea.

Absolutely.  I don't think it should be mandatory, I just haven't
seen those arguing against it come up with another solution which
is easy enough to implement (client and server) that it has gained
serious traction.  Meaning that end-users are still in multi-protocol
double-bandwidth-use hell.

Bron.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 22 Jul 2011 12:52:42 +0200

On 07/22/2011 01:39 AM, Randall Gellens wrote:
> Oh, that wasn't a problem. There were more serious security issues (from
> lashing together different protocols to implementation issues with
> executing other programs) and a complete lack of support for SMTP
> extensions.

Sieves send mail, and appear not to have these problems to a bothersome 
degree. So I tentatively posit that today, an extension can be designed 
well enough to avoid significant pain and complaints.

Arnt

---

From: Gilles LAMIRAL <[email-masked]>
Date: Sun, 24 Jul 2011 02:34:41 +0200

Hello,

> But we used the car, find a place to park, walk inside the supermarket one time (and not two).
> We saved a lot of time and money. Then, inside the supermarket we can talk to the butcher or take fish.

No at all. With Internet the car is your screen and it doesn't care to deal with one 
or more supermarkets, you don't save lot of time and money adressing several
supermarkets or small stores, you save small peanuts. 

> I hope the same will happen for IMAP5: one single protocol for access (write less code, 
> configure less parameters), but access more services.

Easy: just write a Meta-protocol or meta-service or proxy-anything,
accessing each service and make it accessible like you want.  
It will be simpler, faster, respecting the good old standards 
and it will be optional. The Silver Bullet?

Even in supermarkets there are specialists and no one 
can replace each other. It's easier to change one specialist
than the whole supermarket. Don't ask IMAP to become a supermarket.

> I care. Sometimes that 1% traffic is the most expensive traffic you have paid for in the last 3 years. 
> I happened to me last week with my 3G connection and my android.

Be honest, what you paid is Spam and Fun videos.
Don't ask IMAP to solve the payment method of your Internet access,
it's not the good store to solve that issue, ask your 3G connector.

> Wrong. With UID SUBMIT, IMAP servers will become also SMTP clients, not SMTP servers. 
> We just move SMTP the client logic from the user client to the server.
> The user can still have all the notifications he likes via the IMAP channel or via a returning DSN.

Ok. It's what I said elsewhere, you ask IMAP to be a sub-MUA that have to do 
a little of what full-MUAs already master. You ask IMAP to become
a proxy. So you like proxy. Write one.

> A little configuration work for one person (the server sysadmin) will save a lot of work 
> for the user and helpdesk people to configure SMTP on hundred devices. When we used IMAP+SMTP 
> I lost a lot of time at the phone with the user just to discover that... the network he 
> was using is blocking outgoing SMTP (port 25) and sometimes also blocking SMTPS.

And you might know why those ports are blocked: Spam and mail-bombing.
So when IMAP will be able to post messages the next port to be blocked will be 
the IMAP port for exactly the same reasons: Spam and mail-bombing.

> Two channels (IMAP+SMTP) creates a lot of user support requests at your helpdesk service.

It's just marketing incompetence. Just provide products with already configured 
mailboxes, it's what have done all Internet access providers, they now provide
Internet access via boxes already or automatically configured.
Ask your bosses to think.

> Three channels (IMAP+SMTP+*DAV) means more user support again, because http/https could also be blocked, 

So you may be talking about another thing than Internet access here.

> transparent proxyed and many other horrible things. A *LOT* of user support at the phone.

This is not IMAP related issues.

> This is one of the points because my IT chiefs decided to move to Blackberries or 
> ActiveSync in the last 5 years. End user configuration and support !

So be happy to leave IMAP, why are you arguing?

---

From: Giovanni Panozzo <[email-masked]>
Date: Sun, 24 Jul 2011 11:56:02 +0200

> and it will be optional. The Silver Bullet?

In this ML we are talking about standards, not of my own, self written, 
I use only for me and my customers, proxy.
(should I also write my proxy client for thunderbird, K9, outlook, 
windows live mail, Mail.app, iOS and all the imap clients I know of ???)

> Be honest, what you paid is Spam and Fun videos.

Believe me: NO. It was a mix of:
- Try to send a too big attachment with K9 for android.
   K9 does an APPEND to Trash folder of IMAP (100MB gone)
   before doing SMTP
- K9 not implemented ESMTP SIZE, so it tryed to send 40MB
   of ESMTP and then stopped. This happened at least 4 times.
   (I already filed a BUG on K9 ML to add ESMTP SIZE support)
- This wasted all my prepaid weekly paid traffic (250MB)
- and after fnishing prepaid traffic, i payed for extra 10 euros
   just for SMTP retries

Yes, it's a K9 problem and a stupid user (me) problem. But HERE in this 
ML we should discuss for STANDARDS which will would also help to avoid 
these things happens.

The lack of a standard in the correct sequence of events (APPEND berfore 
SMTP or SMTP before APPEND) generates small confusions, helpdesk 
requests, human errors.

> a proxy. So you like proxy. Write one.

Mmm... so "google webmail" is a proxy ? I think it's too simplistic to 
call it "a proxy".

And as I said above, we are discussing of standards and I'm here to try 
  to make people's life easier. My "proxy" will be only an extra piece 
of software to mantain, debug, fix in the long chain from the user to 
the mail storage. Which will never be able to speak with ohter clients 
or servers.

> the IMAP port for exactly the same reasons: Spam and mail-bombing.

SPAM is mostly based on unauthenthicated SMTP. IMAP requires user 
autentication. So it's very simple to find the sender and block its 
account or report to the police.

> Ask your bosses to think.

Again: in this ML we are talking about standards. 3 differents ISP will 
produce 3 different incompatible boxes. This will not let the user to 
freely switch e-mail provider.
And e-mail is not only the ISP one, but also internal corporate servers, 
for both big corporate and small offices.

> So you may be talking about another thing than Internet access here.

Really I don't known where you work or live, but here at home I can have 
FULL internet access. Inside corporate LANS I have LIMITED access. 
LIMITED means different forms of limitations (sometimes closed ports, 
sometimes https/http traffic is content inspected and filtered...)
A client which can do everything on a single channel has a bigger 
chanche to work than a client that requeres three different channels.

> So be happy to leave IMAP, why are you arguing?

Again: I'm writing in this ML in hope for STANDARDIZATION, because I 
think that a new but more complete standard will make people life easier 
than having many different one (BB vs ActiveSync vs SyncML vs 
SMTP+IMAP+CalDAV+CardDav+Sieve...).

---

From: Bron Gondwana <[email-masked]>
Date: Sun, 24 Jul 2011 13:16:55 +0200

> it's not the good store to solve that issue, ask your 3G connector.

And this attitude is why shit like Skype succeeds over a whole pile
of open alternatives.  Because they solve people's problems, while
everyone else says "blame your ISP for not giving you a fixed IP
address, go configure your router to forward ports, go choose one
of a bunch of random providers and figure out their signup process
and how to connect to them".

Skype says "what username would you like, what password would you
like, optionally fill in some details so your friends can find you
easier" - congratulations, you're connected.

Now, here's a test you can use to make sure all your stuff outside
our control (sound, video, etc) is working correctly.  30s later,
congratulations, you're up and running.  Now would you like to search
for some people you know?  5 minutes and you're video chatting to a
friend.

That's what most people want - to be able to chat to the friend.  If
we're busy saying "no, we won't solve your problem for you, and we
don't think your problem is the kind of problem that smart informed
people should have anyway" then people flock elsewhere.

Why do I want to stay with IMAP?  Because it's the closest to a
decent non-realtime communication tool out there.  It gives pretty
good reading access to email, and a quite usable centralised store
that you can still cache locally and access fast.

A well set up IMAP account is accessible from your desktop, your
webmail and your phone (decent Android software pending) and an
action taken on any of them is reflected on the others.

All the competitors in this space suck more, and I kind of hoped
that in the 10 years I've been caring about this that someone would
have solved the bits of IMAP that suck for me and for most of the
people _I_ deal with.  They might not be the same ones you deal with,
but in years of working Uni helpdesk, running the email systems for
a company and now running an email service for regular users, client
configuration and SMTP injection being on a separate channel have
been one of the biggest headaches.

A massive bonus of doing submission down exactly the same channel
is that most of the time either nothing is working, or everything is
working.  A large source of support headaches is one of IMAP or
SMTP being unavailable for whatever reason, and the other being
available - confusing users when half their actions succeed and
the other half fail.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 24 Jul 2011 23:13:54 -0700 (PDT)

On Sun, 24 Jul 2011, Bron Gondwana wrote:
> of open alternatives.

Skype is a proprietary system, with complete control over all agents that
use it.  It's easy to make things work if nobody else can write an
implementation, and all users use the products that you produce.  You
don't have to worry about some cretin producing an implementation that
doesn't interoperate and falsely blaming your implementation for the
problem.

All you have to do then is convince users to use your system.  That does
mean that you have to have enough dough to fund the development costs for
a complete suite of implementations for a variety of platforms.

That sounds like the type of development environment you would be happiest
in.  Send your resume to Microsoft.

> easier" - congratulations, you're connected.

Skype can do this because there is NO other way to use Skype except for
the Official Skype Way.  Skype doesn't have to offer extensions or
facilities for other implementors; if the official Skype implementation
does not do it, it can't be done.

There are huge simplifications possible when this is done.

But even then, it doesn't always work.  Getting Skype to work on Linux
(and to a lesser extent, Windows) can be a major headache because the
audio and/or video systems have unpredictable configuration issues
depending upon what hardware and driver software is installed.

It's not so hard on Mac since Apple simply doesn't let you put arbitrary
hardware on a Mac...if Apple doesn't have a driver for it then lotsa luck
getting one.

> decent non-realtime communication tool out there.

In that case, you should respect the judgement of the people who made it
that way.

> All the competitors in this space suck more

Then why do you sing their praises?

Have you considered why they suck?

Have you considered that making IMAP be like them would make IMAP suck
like them?

> been one of the biggest headaches.

Then build a proxy, and a client for your proxy.

The exercise would be a good educational experience for you.  It will
teach you why your ideas are wrong-headed, and do so without causing harm
to everything/everybody else.

-- Mark --

http://panda.com/mrc

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 25 Jul 2011 08:56:46 +0200

> doesn't interoperate and falsely blaming your implementation for the

Test suites are good for that, so the theory goes.  It seems to be slowly
working in the web browser space, though they have a much greater
challenge of incompatible implementations to work through.  But the
existence of test suites that spit out a "score" at the end is causing
competition to be correct.

> Then why do you sing their praises?

I'm not singing their praises, I'm observing their success and the
demand for them from our customers.  One of the most common requests
we get from customers is integrated calendaring and "ActiveSync".
They don't necessarily want a specific protocol, they want something
which solves their problems.  They only know of that protocol because
they see other people using it, and stuff seems to just work for them.

> Have you considered why they suck?

Apart from the fact that they were only ever designed around one
implementation?

> Have you considered that making IMAP be like them would make IMAP suck

Yes I have.  But I don't see that having a way to fast resync a folder
after it's been renamed, or to move a bunch of messages in one go
without hitting quota limits is "making is suck".  Deliver is slightly
more interesting, but I still think this is a Hurd vs Linux dilema, and
I'm firmly on the Linux side of that argument.  Microkernels may be
cleaner in theory, but 22 years later I think it's pretty clear that
it failed to produce the theoretically expected results.

> to everything/everybody else.

I already did that for COMPRESS - though that's an admittedly very simple
case.

I can tell you right now that building a proxy for MOVE would be an
exercise in serious fuckedupedness, since the whole point is that you
avoid the quota limitations that mean users get "over quota" when trying
to move messages due to the design of the underlying store, which is
non-intuitive and actively hostile to their expectations.  A proxy
would have to either have the ability to double the quota for the
duration of the move, or talk an enhanced-IMAP[tm] to the backend so
that the quota could be disabled for this action.

The deliver stuff could of course be handled by a proxy, no problem
at all - either at append time or later pulling a message from the
IMAP store and dumping it to sendmail.  Either way we have an
authenticated user to pass to '-f' to get the envelope sender
correct.

And if the proxy had enough local storage, it could emulate the
DIGEST and GUID stuff too.  Obviously it's much more expensive to the
user to calculate the SHA1 (or whatever) on a thousand messages as
they get fetched for the first time than it is to calculate them on
delivery, but an aggressive enough proxy could iterate over all the
mailboxes on the server doing a STATUS call and then fetch anything
new on a regular basis, so only the most recent messages needed to
be digested.

As it is - we're already building a JSON API which wraps a
QRESYNC/CONDSTORE enabled IMAP server along with a bunch of
extra stuff that we've added into Cyrus.  So far it's only used
by our web interface, but over time it will be extended into
other clients Opera is involved with.  Due to the complexity of
MOVE (see above) it doesn't do the quota dance, but it already
handles the other bits and pieces, including GUID based caching
which catches renames more cleanly.

Bron.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 24 Jul 2011 22:48:14 -0700 (PDT)

On Sun, 24 Jul 2011, Giovanni Panozzo wrote:
> Yes, it's a K9 problem and a stupid user (me) problem.

K9 mail sucks in multiple ways.  It sucks so hard it is barely usable.
K9 is filled with wrong-headed design and programming decisions that defy
specific text in multiple standards documents....not to mention common
sense.  The authors of K9 think that they know better and can just ignore
what they don't like in the standards.

So why do so many Android users (including me) use K9 mail?  Because it
sucks less massively than the other choices for Android, especially the
utterly unusable default mail program from Google.

I have been begged(!!) to produce a good email client for Android.  As I
said before, I scoped out the project (including the people I wanted on
the team) and concluded that I needed at least a million dollar budget to
do it.  The ROI isn't there.

> But HERE in this ML we should discuss for STANDARDS which will would
> also help to avoid these things happens.

It is absolutely impossible for ANY open standard to do that.

Only closed standards, where a single entity (e.g., Microsoft) has
absolute dictatorial power, have a chance of such control.

Microsoft can say what is ActiveSync and what is not.  Microsoft can
punish any other vendor of ActiveSync products by making that vendor's
products cease to work...and Microsoft will always be in the right becuase
Microsoft owns ActiveSync.  Microsoft even does this to vendors that they
aren't trying to punish.  Everybody in ActiveSync dances to Microsoft's
tune, and Microsoft frequently changes the beat and the melody.

Nobody has such power in open standards.

> SMTP+IMAP+CalDAV+CardDav+Sieve...).

You are in a drug-induced fantasy, and have utterly missed the point of
open standards.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Sun, 24 Jul 2011 22:14:29 -0700 (PDT)

> a proxy. So you like proxy. Write one.

You are quite correct in this observation.  But there is more to it; these
guys want to pretend that they are not using a proxy.  It is therefore
quite important to them that the "standard way" is their way of proxy (it
would be a disaster for them for it to be someone else's proxy).

> the IMAP port for exactly the same reasons: Spam and mail-bombing.

It's even worse than that.  Consider environments where deep packet
inspection is done on all SMTP traffic at the router for policy reasons,
and TLS is blocked.

> Ask your bosses to think.

It's usually not the boss.  It's the lazy young programmer who only knows
cookbook JAVA techniques.  When something doesn't fit into a cookbook, he
demands that it change rather than learning how to cook without a
cookbook.

> So be happy to leave IMAP, why are you arguing?

Every IT department that goes to ActiveSync means more money to me as
departments set up their own IMAP infrastructures, independent of the IT
department, in order to have email that works.

I owe personal thanks to several such IT departments for all the new
business they have given me.

As for BlackBerry...talk about a dying technology!

-- Mark --

http://panda.com/mrc

---

From: Chris Newman <[email-masked]>
Date: Mon, 25 Jul 2011 14:37:15 -0400

--On July 19, 2011 8:49:15 +0100 Dave Cridland <[email-masked]> wrote:
> wrapper around ESMTP/Submission that it offers no clear benefit, except
> for the ability to avoid BURL/GENURLAUTH. I think it'd be pretty ugly,
> but it *is* possible.

I also believe this *is* possible. However, real SMTP servers have to make 
auditing and access control decisions based on the source IP address, user 
authentication, etc, to stop spam/etc from both internal and external 
sources. So the IMAP server becomes a proxy to the SMTP server that has to 
forward all that information to the SMTP server. So we actually need an 
SMTP extension that the IMAP client can use to forward that information. 
What happens if we don't do that? Well, real SMTP deployments need 
rate-limiting controls based on the source address, even if the source user 
is authenticated (Windows worms install botnets that use Outlook mail 
credentials for the user to send spam and phishing). So the entire IMAP 
server will likely get marked as a spam source once any user abuses it, 
unless there's an SMTP extension for the IMAP server to use to forward 
credentials and source IP.

The problem is SMTP may _appear_ easy, but a real implementation and 
deployment of an SMTP server is in many ways more subtle and complex than 
an IMAP server. Implementing all the spam, virus, phishing, rate-limiting, 
etc features in an IMAP server codebase is a huge architecture error.

> Now, you absolutely could just solve the minimal case, but that also has
> the effect of closing the door on all future development, so I'm less
> than convinced that's a workable idea, either.

It also breaks some types of spam/virus prevention and denial-of-service 
defenses for outbound mail.

> The trick is *not* covering 100% of the existing submission cases; the
> trick is including sufficient extensibility to make 100% parity possible.

Two IMAP commands plus an SMTP command could do it. If we choose a 
reasonable minimum-required-set of SMTP extensions then most clients 
wouldn't need to use the GETEHLO command (but it's critical for future 
extensibility).

IMAP command 1: GETEHLO
IMAP command 2: SEND [(ALSOAPPEND <mailbox>)] 
<entire-SMTP-envelope-with-params> <body>
SMTP command: PROXYHELO <about-10-parameters-needed>

If someone is willing to do all the editorial work necessary to correctly 
specify commands like this, and document the security implications, then I 
would not object to such an effort (and I would do my best to handle the 
PROXYHELO command and security implications well). I was a strong supporter 
of the trio proposal because it was more flexible and I thought the 
security implications would be simpler. After working on a 
BURL/URLAUTH/CATENATE implementation, I am no longer convinced the latter 
is the case -- the complexity of the security implications of both 
proposals are roughly equivalent but live in different places.

		- Chris

---

From: "Bron Gondwana" <[email-masked]>
Date: Tue, 26 Jul 2011 16:24:19 +0200

On Mon, 25 Jul 2011 20:37:15 +0200, Chris Newman <[email-masked]>  
wrote:

> I also believe this *is* possible. However, real SMTP servers have to  
> make auditing and access control decisions based on the source IP  
> address, user authentication, etc, to stop spam/etc from both internal  
> and external sources.

True - but at least in the case of Cyrus, which has a full sieve
implementation, the user can already cause email to be generated
by creating a sieve script with a forward rule.  The IMAP server
already needs to know how to inject email into an outbound queue.

> So the IMAP server becomes a proxy to the SMTP server that has to  
> rate-limiting controls based on the source address, even if the source  
> user is authenticated (Windows worms install botnets that use Outlook  
> mail credentials for the user to send spam and phishing). So the entire  
> IMAP server will likely get marked as a spam source once any user abuses  
> it, unless there's an SMTP extension for the IMAP server to use to  
> forward credentials and source IP.

Forwarding source IP is actually pretty tricky right through the
various layers - the usual way seems to be following the Received
headers back to the first "untrusted" host.  It would seem that
having the IMAPd put a received header at the top of the message
when it injected it would fulfill this purpose.

> deployment of an SMTP server is in many ways more subtle and complex  
> than an IMAP server. Implementing all the spam, virus, phishing,  
> rate-limiting, etc features in an IMAP server codebase is a huge  
> architecture error.

You have me slightly convinced by your argument.  But only slightly,
because...

> defenses for outbound mail.

... how is that different to webmail, which also has to inject email  
somehow.
Not very.  All the people advocating webmail as the solution face exactly
the same problems.  And they do much the same thing - inject a header with
the original IP address.

We do outbound rate limiting and spam scanning and all that good stuff,
and we do it by tracking the original authenticated user who submitted
the email.  We don't also track which IP address it came from once the
user is authenticated, because generally attackers have more IPs than
they have valid user accounts, so it's pointless.

I don't think that's likely to change, particularly with ipv6 where you
can have a few million distinct IPs to play with if you want.

> SMTP command: PROXYHELO <about-10-parameters-needed>

the ALSOAPPEND bit is racy.  I'm 99.9% convinced (here I go with
magic numbers again) that you're better off APPENDing separately,
and then using the message in the spool once that has succeeded.
If the rest fails, you can always EXPUNGE the message again.

Which is basically what APPEND/CATENATE/BURL stuff does, except
that the user authentication is only managed through one channel.
I still think the "SUBMIT" part should be to a local delivery
queue rather than an SMTP server, but maybe that's my *nix
background talking, where every sane machine has a 'sendmail'
compatible wrapper which does the right thing.

Bron.

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Gilles LAMIRAL <[email-masked]>
Date: Tue, 26 Jul 2011 17:12:54 +0200

Hello,

> ... All the people advocating webmail as the solution face exactly
> the same problems.  And they do much the same thing 

No, webmail writers do what they want, they are not following a RFC that have
to work for twenty years.

---

From: Randall Gellens <[email-masked]>
Date: Tue, 26 Jul 2011 11:10:47 -0700

At 4:24 PM +0200 7/26/11, Bron Gondwana wrote:

>   The IMAP server

Inject into an outbound queue or submit?  There's a difference -- 
submission is much earlier in the process.

-------------- Randomly selected tag: ---------------
The earth is like a tiny grain of sand, only much, much heavier.