Subject: [ietf-smtp] Submission DATA reply to indicate auto-save

From: Timo Sirainen <[email-masked]>
Date: Thu, 11 Apr 2013 00:22:10 +0300

GMail's submission server saves all the submitted mails to the sender's Sent mailbox. After this most IMAP clients with their default configuration will again save the same mail to the Sent mailbox. I'm not sure if GMail removes one of them as a duplicate or not. I've been considering doing a similar feature to Dovecot (it will most likely have a proxying submission server in near future), but it would be nice to tell the SMTP clients that this auto-saving is happening, so they could skip the IMAP APPEND part with default configuration.

Of course there are the LEMONADE extensions that are supposed to solve this same issue, and Dovecot will support those as well, but I think it's going to take a while for clients to actually implement those. There's likely much better chance of them implementing a small check for "oh, the mail was already saved, I'll just skip the IMAP APPEND." Even if any clients themselves didn't implement this, it would make it possible to implement a simple SMTP+IMAP proxy locally that catches the flag and supresses the APPEND before it reaches the network.

So, any thoughts on creating an RFC for this functionality? I'm not familiar enough with SMTP yet to know what would be the best design for that notification. Maybe a new extended status code? Maybe change some of the DATA reply's text part to have a special meaning? A simple flag is enough I think, because it's reasonable to require that both the server and client supports IMAP SPECIAL-USE to find the \Sent flagged mailbox. Returning IMAP UIDVALIDITY+UID could help IMAP clients a bit more, but that's maybe a bit too ugly (and not always possible).

---

From: SM <[email-masked]>
Date: Wed, 10 Apr 2013 15:40:53 -0700

Hi Timo,
At 14:22 10-04-2013, Timo Sirainen wrote:
>GMail's submission server saves all the submitted mails to the 
>sender's Sent mailbox. After this most IMAP clients with their 
>default configuration will again save the same mail to the Sent 
>mailbox. I'm not sure if GMail removes one of them as a duplicate or

I assume that you have read Gmail's answer about the handling of the 
/Sent mailbox.

>  not. I've been considering doing a similar feature to Dovecot (it 
> will most likely have a proxying submission server in near future), 
> but it would be nice to tell the SMTP clients that this auto-saving 
> is happening, so they could skip the IMAP APPEND part with default 
> configuration.
>
>Of course there are the LEMONADE extensions that are supposed to 
>solve this same issue, and Dovecot will support those as well, but I 
>think it's going to take a while for clients to actually implement 
>those. There's likely much better chance of them implementing a 
>small check for "oh, the mail was already saved, I'll just skip the 
>IMAP APPEND." Even if any clients themselves didn't implement this, 
>it would make it possible to implement a simple SMTP+IMAP proxy 
>locally that catches the flag and supresses the APPEND before it 
>reaches the network.
>
>So, any thoughts on creating an RFC for this functionality? I'm not 
>familiar enough with SMTP yet to know what would be the best design 
>for that notification. Maybe a new extended status code? Maybe 
>change some of the DATA reply's text part to have a special meaning? 
>A simple flag is enough I think, because it's reasonable to require 
>that both the server and client supports IMAP SPECIAL-USE to find 
>the \Sent flagged mailbox. Returning IMAP UIDVALIDITY+UID could help 
>IMAP clients a bit more, but that's maybe a bit too ugly (and not 
>always possible).

In the above you have the SMTP (MSA) client and the IMAP client which 
are acting independently.  If the mail client already knows that the 
message has been saved to the Sent mailbox it could use the
Message Submission BURL Extension to submit the message to avoid the 
"duplicate submission".

Suppressing the APPEND may be problematic.  The MSA server can reject 
the submission attempt with a message mentioning the IMAP mailbox 
automatically used to submit the message.  BTW, what you are looking 
for might be what behavior the mail client should set at 
configuration time to provide mail functionality.  This means 
detecting the features offered on the server side.

Regards,
-sm

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 11 Apr 2013 02:33:13 +0300

On 11.4.2013, at 1.40, SM <[email-masked]> wrote:

>> GMail's submission server saves all the submitted mails to the sender's Sent mailbox. After this most IMAP clients with their default configuration will again save the same mail to the Sent mailbox. I'm not sure if GMail removes one of them as a duplicate or
> 
> I assume that you have read Gmail's answer about the handling of the /Sent mailbox.

Not really. I've seen some random mails that GMail doesn't do any duplicate prevention, but somehow I thought it there would be something.. at least in theory there could be.

>> not. I've been considering doing a similar feature to Dovecot (it will most likely have a proxying submission server in near future), but it would be nice to tell the SMTP clients that this auto-saving is happening, so they could skip the IMAP APPEND part with default configuration.
>> 
> In the above you have the SMTP (MSA) client and the IMAP client which are acting independently.  If the mail client already knows that the message has been saved to the Sent mailbox it could use the
> Message Submission BURL Extension to submit the message to avoid the "duplicate submission".
> 
> Suppressing the APPEND may be problematic.  The MSA server can reject the submission attempt with a message mentioning the IMAP mailbox automatically used to submit the message.  BTW, what you are looking for might be what behavior the mail client should set at configuration time to provide mail functionality.  This means detecting the features offered on the server side.

My understanding is that most email clients first send the message to submission server, and only after that is successful they are saved to Sent mailbox via IMAP APPEND. My whole auto-save-flag idea relies on that..

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Thu, 11 Apr 2013 00:44:40 +0100

On 10 Apr 2013, at 22:22, Timo Sirainen wrote:
> sender's Sent mailbox. After this most IMAP clients with their default 
> configuration will again save the same mail to the Sent mailbox. I'm 
> not sure if GMail removes one of them as a duplicate or not. I've been 
> considering doing a similar feature to Dovecot (it will most likely 
> have a proxying submission server in near future), but it would be 
> nice to tell the SMTP clients that this auto-saving is happening, so 
> they could skip the IMAP APPEND part with default configuration.

Yes, GMail suppresses duplicate messages across the entire store, 
apparently solely by use of the Message-ID field, with very interesting 
results (including, for example, that one never receives one's own 
postings to mailing lists).

Just off the top of my head, I can't see how an SMTP extension would 
serve much more than an advisory purpose; either the client understands 
the advertised extension and reacts appropriately, or it does not and 
then you must process the duplicate received via IMAP yourself.  The 
SMTP server could, I suppose, internally relay client information to the 
IMAP server, but then you're back to the inherent difficulties of the 
BURL solution, because the client must explicitly request special 
processing for the SMTP server.  For this to be simple and controlled by 
the client, the information is from server to client, so you would have 
to handle the existing "Client duplicates" case regardless.  This is 
still fine, I suppose, because it costs less for you to discard a 
duplicate than it does for a client to retransmit it.  Still a bit 
unclean, though. :-)

Cheers,
Sabahattin

---

From: Tony Finch <[email-masked]>
Date: Thu, 11 Apr 2013 01:12:06 +0100

The proper ESMTP style for this would be

(1) an EHLO keyword to say the server supports the feature
(2) a MAIL parameter for the client to enable the feature

Do you need more signalling than that? I think adding syntax to the message body reply would be a bad idea, since that is usually used to feed back trace info (queue ID). Perhaps you could have a new command to be used (pipelined) after the body to retrieve the IMAP status. Would that make the MAIL parameter redundant?

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Apr 2013 20:59:54 -0400

--On Thursday, April 11, 2013 01:12 +0100 Tony Finch
<[email-masked]> wrote:

> Do you need more signalling than that? I think adding syntax
> to the message body reply would be a bad idea, since that is
> usually used to feed back trace info (queue ID). Perhaps you
> could have a new command to be used (pipelined) after the body
> to retrieve the IMAP status. Would that make the MAIL
> parameter redundant?

Yes.  But it seems to me that the problem Timo is trying to
solve involves submission via Gmail and not being able to turn
off its saving a copy of the sent message.   It seems to me that
problem doesn't involve the IMAP server at all except insofar as
the sending client chooses to send the SMTP message and then use
IMAP APPEND or equivalent to put a copy somewhere.

It seems to me that problem is easily solved using the
well-known maxim:

	Patient: "Doctor, it hurts when I do xyz."
	Doctor:  Have you considered not doing xyz?"

Option 1: Persuade the Gmail folks to allow a profile
	flag that would _not_ save everything to the /Sent
	folder but would allow explicit copies to it.
	
Option 2: Fix the relevant client to support a
	per-submission-server flag that determines whether or
	not to append copies of outgoing messages to some
	selected folder.

However easy or difficult those options might be, they have the
advantage of requiring no protocol changes and hence no updating
of clients and servers to handle a new dialogue and options.
FWIW, I note that Mulberry, which is getting long in the tooth
in other respects, has been easily configured to support Option
2 for as long as I can remember.

Or am I missing something here?

    john

---

From: Dave Crocker <[email-masked]>
Date: Wed, 10 Apr 2013 18:52:07 -0700

wfm.

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: SM <[email-masked]>
Date: Wed, 10 Apr 2013 23:23:52 -0700

Hi Timo,
At 16:33 10-04-2013, Timo Sirainen wrote:
>Not really. I've seen some random mails that GMail doesn't do any 
>duplicate prevention, but somehow I thought it there would be 
>something.. at least in theory there could be.

Their recommendation is for the mail client not to save sent messages 
to the (IMAP) server.

>My understanding is that most email clients first send the message 
>to submission server, and only after that is successful they are 
>saved to Sent mailbox via IMAP APPEND. My whole auto-save-flag idea 
>relies on that..

The idea of suppressing the IMAP APPEND may work.  I would not 
suggest doing it as it can affect other functionality that is not 
currently available.

Regards,
-sm

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 11 Apr 2013 12:07:01 +0300

On 11.4.2013, at 3.59, John C Klensin <[email-masked]> wrote:

>> Do you need more signalling than that?

Gmail is unlikely to implement that, since that would require changing their current default behavior and breaking users' configuration. So it's not going to get wide adoption.

EHLO advertisement itself isn't enough, since this may be a user-specific setting and I don't think clients re-EHLO after AUTH.

Something that could work is something like:

C: EHLO ..
S: 250-LOCALCOPY
S: ..
C: AUTH ..
S: ..
C: MAIL FROM <..> (LOCALCOPY)
S: 250-LOCALCOPY
S: 250 2.1.0 Ok

Where the above server notifies that the user will get a local copy, while:

..
C: MAIL FROM <..> (LOCALCOPY)
S: 250 2.1.0 Ok

Here the server doesn't store a local copy. Except if the local copy saving fails (e.g. user over quota), should the server now also fail the DATA? So maybe instead:

C: MAIL FROM <..> (LOCALCOPY)
S: 250 2.1.0 Ok
..
C: DATA
C: ..
S: 250-LOCALCOPY
S: 250 2.0.0 Ok: queued as 6AA4E3C6224

That would also allow more complexity like:

250-LOCALCOPY 2.0.0 Ok
250-LOCALCOPY 5.2.2 You are over quota
250-LOCALCOPY 2.0.1 IMAP 12345678 123 Saved mail with IMAP UIDVALIDITY 12345678 and UID 123 to your \Sent mailbox

>> I think adding syntax

I guess the 250-LOCALCOPY replies could also be a result for a new command. Although that could be more difficult for clients to implement using standard SMTP libraries. But then again the MAIL FROM parameter might not be possible either..

> 	folder but would allow explicit copies to it.

But that feature is actually useful, if only the clients realized it was happening. That's why I'm considering adding such option myself optionally.

> 	not to append copies of outgoing messages to some

Annoying to require users to configure a setting that could be so easily determined automatically. Most of the users aren't going to configure that, so the feature would be of no use for them. I'm hoping to build something that would get wide adoption on server side (Gmail, Dovecot) and hopefully on client side ("needs about 10 lines of more code to client" - that's why I would have liked a simple flag on DATA reply, clients already do everything necessary to simply parse that reply and set an internal flag to skip IMAP APPEND).

---

From: Tony Finch <[email-masked]>
Date: Thu, 11 Apr 2013 10:31:41 +0100

Timo Sirainen <[email-masked]> wrote:
>
> Gmail is unlikely to implement that, since that would require changing
> their current default behavior and breaking users' configuration. So
> it's not going to get wide adoption.

Always a problem with protocol upgrades :-) I wonder if it makes sense to
add a parameter to the EHLO keyword giving the default, e.g. GMail would
say:

250-LOCALCOPY YES

and the client could state its preference which overrides the default:

MAIL FROM:<...> LOCALCOPY=YES
MAIL FROM:<...> LOCALCOPY=NO

I wonder how to handle per-user localcopy settings. I think you would need
a command to query in that case. e.g. an EHLO reply like

250-LOCALCOPY USER

The LOCALCOPY command could retrieve both the option's state and the
disposition of the last MAIL transaction, if there was one.

> 250-LOCALCOPY 2.0.1 IMAP 12345678 123 Saved mail with IMAP UIDVALIDITY 12345678 and UID 123 to your \Sent mailbox

Ugh I don't like the mixture of 250 5.2.2 :-) There is the potential to
fix a long-standing nastiness, which Lemonade missed: a good
implementation could ensure that message submission and saving a local
copy either both fail or both succeed.

Speaking of Lemonade, I wonder about combining with BURL ...
MAIL FROM:<...> LOCALCOPY=MOVE ?
Getting into dangerous over-engineering territory.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

---

From: John C Klensin <[email-masked]>
Date: Thu, 11 Apr 2013 11:07:23 -0400

--On Thursday, April 11, 2013 12:07 +0300 Timo Sirainen
<[email-masked]> wrote:

>...
 
> Gmail is unlikely to implement that, since that would require
> changing their current default behavior and breaking users'
> configuration. So it's not going to get wide adoption.

I wasn't suggesting changing the current default behavior at
all.  Instead, I was suggesting that they could add a new and
thoroughly optional flag to the profile that would default to
the current behavior but that would, if set by the user, disable
the copying of the outgoing message.  Current users and users
favored the copies wouldn't need to do a thing. 

I think you've missed my point.  I think that what you are
looking for is a configuration option on either the client or
the server, not a protocol matter.  Trying to make a protocol
change to avoid configuration options -- especially when that
protocol change has to affect both SMTP and IMAP in the general
case-- is unwise architecturally and philosophically.  More
important from your point of view, if you don't believe Google
would implement a configuration flag that could disable the
message-copy behavior, why is it reasonable to believe that they
would implement a protocol change?  

Any attempt to actually negotiate the server behavior during the
mail transaction would presumably have worse performance
implications than a per-user configuration setting.

> EHLO advertisement itself isn't enough, since this may be a
> user-specific setting and I don't think clients re-EHLO after
> AUTH.

Continuing with my reasoning above, if it is a user-specific
setting, then the server configuration option I suggested is
already in place and the user presumably knows it is.  If the
configuration option doesn't exist, then there are no
user-specific settings.
 
> Here the server doesn't store a local copy. Except if the
> local copy saving fails (e.g. user over quota), should the
> server now also fail the DATA? So maybe instead:
> 
> 250-LOCALCOPY 2.0.1 IMAP 12345678 123 Saved mail with IMAP
> UIDVALIDITY 12345678 and UID 123 to your \Sent mailbox

See above.  The more complex this gets and the more message-time
handshaking is required, the less likely it is to be
implemented.  Worse, a given Gmail (or other) mailbox may be
accessed over webmail, IMAP, POP, and maybe by other means,
using different clients with different capabilities.  Some of
the above might require that the client send the
I-only-use-IMAP-and-wont-change-my-mind-in-the-future option and
that the server believe it.

>...
> I guess the 250-LOCALCOPY replies could also be a result for a
> new command. Although that could be more difficult for clients
> to implement using standard SMTP libraries. But then again the
> MAIL FROM parameter might not be possible either..

Probably possible.  But whether server and client implementers
would judge it to be worth the trouble is another matter.
Again, if, as you suggest, Google (for example) couldn't be
motivated to implement a rather simple configuration option, why
should one believe that they would implement something as
complex as what you are suggesting.

>...
> But that feature is actually useful, if only the clients
> realized it was happening. That's why I'm considering adding
> such option myself optionally.

So the user of that client program either

-- doesn't know and presumably doesn't care about the
	extra copies
-- does know and still doesn't care
-- knows and wants to avoid the duplicates by
	suppressing the automatic server copy
-- knows and wants to avoid the duplicates by not making
	a copy in the client via IMAP APPEND

If the user doesn't know, then anything one might do with the
options you suggest would be based on the assumption that you
know better than the user does what the user would like if he
were better informed. That is a dangerous assumption and
behavior even when you get it right.  If the user does know,
then the third case is best dealt with by a configuration option
to the server account and the fourth one by a configuration
option to the client program.  And those options may be
per-server --if I had two servers, operated by two different
parties, that had the option of automatically keeping copies of
outbound messages, I might want to set that option to "no" on
one of them but leave it as "yes" on the other, especially if I
had bad experiences of the reliability of the associated IMAP
server in doing the APPEND.

Remember that, for an IMAP user, APPEND offers the choice of
determining what folder the outgoing message going into.  That
determination could be made manually or based on the outgoing
address(es), etc.  Google's "save automatically" cannot
reasonably support that unless you take your command sequence a
step further and send "and when you store it, put it in folder
XYZ" (or, in Gmail's case, "attach label XYZ").   If you build
an automatic "if the server is saving automatically, don't save
locally" capability, you deprive at least some users of that
folder-selection capability.

> Annoying to require users to configure a setting that could be
> so easily determined automatically.

But it can't be "determined automatically" unless you are
certain that you know what the user would want in all of the
many situations that might apply, especially if the IMAP server
and mailstore were physically supported by different machines
and network configurations than the SMTP one.

> Most of the users aren't
> going to configure that, so the feature would be of no use for
> them. I'm hoping to build something that would get wide
> adoption on server side (Gmail, Dovecot) and hopefully on
> client side ("needs about 10 lines of more code to client" -
> that's why I would have liked a simple flag on DATA reply,
> clients already do everything necessary to simply parse that
> reply and set an internal flag to skip IMAP APPEND).

Interesting.  In my experience, many clients have no idea at all
what to do with information that comes back in the DATA reply.
Worse, their strongly interpreting that information violates the
5321 prohibition on using more information than is present in
the reply code.  You could override that behavior with an
extension, but that adds even more complexity and contradicts
"already do everything necessary...".

I don't know that what you are suggesting is a bad idea.  It is
just something I'd think about in a different way and that I
think you'd have a lot of trouble getting implemented and
deployed in precisely the mail implementations that you care
most about (other than your own, of course).

best,
    john

---

From: Paul Smith <[email-masked]>
Date: Thu, 11 Apr 2013 16:25:07 +0100

On 11/04/2013 16:07, John C Klensin wrote
> favored the copies wouldn't need to do a thing.
The problem is always that users are thick, so automating it would be a 
good thing

Personally, I'd go for a simple and crude method - have a standardised 
response in the 250 after DATA. If the client understands it, you win. 
If it doesn't, you lose nothing over the current situation

250 2.0.0 Ok: queued as 6AA4E3C6224 [copiedto:sent]

No negotiations needed, trivial server-side programming, and simple client-side programming.

If the server was going to support a more complex method, they'd certainly support an option to turn off server-side copying for people who want that. If the client was going to support a more complex method, it can handle just skipping the IMAP APPEND stage.

-

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 11 Apr 2013 22:10:48 +0300

On Thu, 2013-04-11 at 11:07 -0400, John C Klensin wrote:
> the server, not a protocol matter.  

I tend to agree that this functionality should be configured server side
using some external means (e.g. web UI) instead of doing it in the SMTP
protocol. Otherwise it adds more complexity than necessary. All I really
want is for an SMTP client to know whether the local copying happened or
not.

> Trying to make a protocol

Even if Gmail implemented a configuration flag to disable the behavior,
it wouldn't solve what I'm trying to solve. I like that behavior. I want
to make it more popular, not less popular. To do that would require
making it easy for clients to support it.

If the total amount of code to change in both server and client side is
just a few lines, and the gain would be to avoid re-uploading the mail,
I think it would quickly get popular. (As opposed to BURL, which
requires a ton of new code and a whole new way of doing the submission.)

> that the server believe it.

Sure, I agree. A single flag in some reply would be ideal solution for
everyone I think, unless it breaks something I didn't realize could
cause breakage.

My IMAP-specific ideas were just thoughts of some optional add-ons,
which wouldn't have to be supported by anyone. I can just as easily live
without them. (Their main purpose would be to allow IMAP clients to add
the message immediately to their local cache, similar to IMAP's UIDPLUS
extension.)

> server in doing the APPEND.

I would definitely make this feature optional on server side, defaulting
to disabled. Each site's admin might want to enable this specifically
for their own users. But the admins wouldn't want to explain to every
user how exactly to configure their 10 different IMAP clients to avoid
the duplicate upload.

> folder-selection capability.

I don't think there are many users who want to save the message anywhere
else than the default Sent mailbox (which is configured by IMAP
SPECIAL-USE extension). Those few who want something different are most
likely able to also disable this feature (or more likely haven't enabled
it in the first place).

> what to do with information that comes back in the DATA reply.

They still do parse the line to see if it's returning 250 or an error.
And most likely they could simply add around that part of the code
something like:

if (strstr(data_reply, "[localcopy]") != NULL) 
  state->imap_append_skip = true;

That was my original plan, but that's a rather IMAP-like way to deal
with it.

> "already do everything necessary...".

I don't think it violates anything, because the client doesn't have to
understand it. Or otherwise enhanced status codes would violate it as
well. Maybe it could be just an enhanced status code (category)?

250 2.8.1 Ok: queued as 6AA4E3C6224, as well as saved locally

But here I don't know if some clients panic because they no longer see
"250 2.0.0" reply.

> most about (other than your own, of course).

The server side implementation is tricky to implement well, but I think
that's mostly out of scope for the potential RFC. I think this extension
would be worth it just for Gmail itself, since they have implemented it
anyway. I was actually a bit surprised that they already didn't indicate
this using some X-extension.

---

From: Bart Schaefer <[email-masked]>
Date: Thu, 11 Apr 2013 14:54:04 -0700

> likely able to also disable this feature (or more likely haven't enabled

Isn't this sort of advice already passed around by a number of email
clients as the Fcc: (file carbon copy) header?

Yes, that header is typically removed by the client before message
submission, but perhaps a reasonable approach here is for the server to
advertise at EHLO that it supports an Fcc extension.  The client can then
choose to (a) include the Fcc to ask the server to store the mail, or (b)
store the mail itself (locally or by IMAP) and omit the Fcc, or even (c)
store the mail itself and also assert something like Fcc: /dev/null to
force the server to discard its copy.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 12 Apr 2013 10:16:24 +0100

Hi Timo,

> But here I don't know if some clients panic because they no longer see

(I have limited Internet connectivity, so might have missed a part of the discussion. My apologies if this was already discussed.) 

If you want to return extra information on DATA, I suggest you put it after the Enhanced Status Code (if any), as many response parsers are expecting Enhanced Status Codes to come first. E.g.:

250 2.0.0 [localcopy] Message accepted

Or

250 [localcopy] Message foo is on its way.