Subject: [ietf-smtp] Fwd: Request to form a new WG: JMAP

From: Alexey Melnikov <[email-masked]>
Date: Mon, 7 Nov 2016 17:31:31 +0000

Hi,

I am forwarding my email here, as I suspect people on these two mailing 
lists might be interested in this.

Please post your comments/questions to [email-masked] or directly to me.

Best Regards,

Alexey

-------- Forwarded Message --------
Subject: 	Request to form a new WG: JMAP
Date: 	Mon, 07 Nov 2016 17:17:59 +0000
From: 	Alexey Melnikov <[email-masked]>
To: 	DISPATCH <[email-masked]>

Hi,
I have received a request to form a new WG: JSON Mail Access Protocol
(JMAP). Draft charter is included below. Please send your comments in
reply to this email or directly to me.

I am also looking for possible chairs for this work, so if you are
interested, please contact me off-list.

Thank you,
Alexey

------------------------------------

Name:    JSON Mail Access Protocol
Acronym: jmap
Area:    Applications and Real-Time Area (art)

Charter for Working Group

Many companies and projects are developing their own JSON based
representations of email which are proprietary, non-standard, and
incompatible with each other. These protocols are proliferating due
to existing standards being insufficient or poorly suited to the
environments they are operating in, particularly mobile and webmail.

The JMAP working group will specify an mechanism to allow clients to
both view and send email from a server over a single stateless HTTPS
channel with minimal round trips. The protocol will support
out-of-band signaling of changes, and will give mobile clients
significant benefits in terms of battery life and network usage.

The use of multiple protocols to perform actions within a single
application creates significant support challenges, as users may get a
variety of partial failure modes (for example, can receive email, but
can not send new messages).  This is further exacerbated if the
different protocols are authenticated separately.

The work of this group is limited to developing a protocol for a client
synchronising data with a server. Any server-to-server issues or
end-to-end encryption of messages are out of scope for this working
group.

The working group will coordinate with the Security Area on credential
management and authentication.

Input to working group discussions shall include:

- Internet Message Format
[RFC 5322]

- CONDSTORE and QRESYNC
[RFC 7162]

- Collection Synchronisation for WebDav
[RFC 6578]

- LEMONADE and experiences from adoption of its output
[https://datatracker.ietf.org/wg/lemonade/charter/]

- SMTP SUBMISSION
[RFC 4409]

- SMTP BURL
[RFC 4468]

The working group will deliver the following:

- A problem statement detailing the deployment environment and
   situations that motivate work on a new protocol for client to
   server email synchronisation.  The working group may choose
   not to publish this as an RFC.

- A document describing an extensible protocol and data structures which
   supports stateless use, flood control and batched operations.

- A document describing how to use the extensible protocol over HTTPS
   with the data structures expressed as JSON.

- A document describing a data model for email viewing, management,
   searching, and submission on top of the extensible protocol.

- An executable test suite and documented test cases to assist
   developers of JMAP servers to ensure they conform to the
   specifications.

---

From: John C Klensin <[email-masked]>
Date: Mon, 07 Nov 2016 12:59:00 -0500

Alexey,

These are just questions, not opposition, but the answers should
probably be clear in any charter or chartering effort.

(1) We've had a long history of variations on The Format to End
All Formats, none of which have lasted a very long time.  I
think JSON is better than most, but, if yet another wonderful
format shows up in a few years and takes over, is it worth
thinking now about what a transition plan would look like?  

(2) Some of us believe, at least on some days, that a key reason
why SMTP, the message formats that started with RFC 822, POP3,
and IMAP have survived against a large number of proprietary and
other SDO alternatives have been precisely because they operate
in plain text, with all but IMAP using very simple
command-argument or name:value syntax.  Are we ready to give
that up?  Do we really need interoperability between a server
and a captive Webmail interface?  Arguably, standards of that
type are part of the province of the IETF only if someone is
contemplating generic webmail clients that can interact with the
servers of more than one organization.

(3) In terms of email models, this would presumably provides an
alternative to IMAP (or maybe POP3) on the mail receiving side.
Would it be equally sensible to define an IMAP extension that
would deliver responses (and maybe accept requests) in a
specified JSON format rather than the LISP-like IMAP one?  Seems
to me that would promote much more compatibility and flexibility.

(4) Similarly, there is lots of practice about use of IMAP to
send.  Is to time to make sure the standards for that
(presumably as an alternative to RFC 6409-style message
submission) and then provide a JSON format IMAP option to
support this form?

Again, just questions, but questions I think we need to ask
carefully before standardizing yet another alternative way to do
much the same thing.

    john

--On Monday, November 07, 2016 17:31 +0000 Alexey Melnikov
<[email-masked]> wrote:

> Hi,
> 
> I am forwarding my email here, as I suspect people on these
> two mailing lists might be interested in this.
>

---

From: Ned Freed <[email-masked]>
Date: Mon, 07 Nov 2016 10:05:06 -0800 (PST)

Without getting into whether or not this is in general a good idea, I want to
note that the proposal as written is self-contradictory, making it very

> environments they are operating in, particularly mobile and webmail.

"Representation of email" would seem to imply that we're talking about a new
message format.

> channel with minimal round trips.

And now we appear to be talking about a replacment for both IMAP and SUBMIT.
This effectively means we're replacing all of the email protocol and format
suite with the exception of SMTP.

> The protocol will support

And now we appear to be talking about server->client synchronization, which
is, in standards terms at least, new.

> different protocols are authenticated separately.

So we're clearly aiming to replace multiple protocols here.

> synchronising data with a server.

Whoops! Now we're talking we're limiting ourselves to synchronization of
the client with the server. Which would seem to exclude SUBMIT, unless
you're doing SUBMIT with magic folders and header fields or something
similar.

> Any server-to-server issues or

If you're talking about a new message format, you're going to have
to deal with end-to-end encryption issues whether you like it or not, either
by changing SMTP as well so that message in this new format can in fact
be transmitted end-to-end, by defining an encapsulation format, by allowing
servers to perform gateways, etc.

> management and authentication.

Again, if you're talking about a new message format, this isn't going
to be remotely adequate.

> - Internet Message Format

This also would seem to imply the message format is in play.

> [RFC 4409]

And SUBMIT is in play.

> [RFC 4468]

Again, I'm not prepared to offer an opinion as to whether or not this whole
thing is a good idea. But I will say that effectively replacing the entire
email protocol suite - which is entirely consistent with what's  been said here
- would at a minimum be a tremendous amount of work requiring a tremendous
amount of justification, and what's in this propsoal doesn't come close.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 07 Nov 2016 13:46:44 -0500

In addition to my earlier comments, I agree with everything in
Ned's note (the two sets of comments seem to be complementary).
Also, please note:

--On Monday, November 07, 2016 10:05 -0800 Ned Freed
<[email-masked]> wrote:

>...

>> [RFC 4409]

4409 has been obsolete for five years now, implying that the
submitters of this proposal have some reading and other homework
to do.   RFC 6409 should be the reference here.

>...

In addition, I'd add to Ned's comment about end to end
encryption the observation that it is particularly important for
webmail and similar models because users or many such services
may have reason to not trust their email service providers (at
one end or the there) with message content.

     john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 7 Nov 2016 19:36:33 +0000

My two cents: Interesting in principle but it has to be compatible with 
what gmail is doing. If the world's biggest site implements one protocol 
and an RFC specifies another, the RFC is forgotten.

Remember the old motto, rough consensus and running code.

Arnt

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 07 Nov 2016 20:57:01 +0000

I wouldn't necessarily read "representation of email" as meaning message 
format.

It could mean representation of email interface.

As for other things, there are plenty of things which could use 
improvement, and I doubt whether it will ever happen in the current IMAP 
framework due to the problem of there being too many complicated 
combinations of optional extensions.

IMAP5 could do something useful, but the last attempts to kick-start 
that were still-born.

Adrien

------ Original Message ------
From: "Ned Freed" <[email-masked]>
To: "Alexey Melnikov" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "[email-masked]'" 
<[email-masked]>
Sent: 8/11/2016 7:05:06 AM
Subject: Re: [imapext] [ietf-smtp] Fwd: Request to form a new WG: JMAP

>_______________________________________________
>imapext mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/imapext

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 7 Nov 2016 22:56:08 +0100

Hi all,

> servers of more than one organization.

My two cents here:  someone proposed 2 years ago a similar model for 
NNTP.  It was named... JNTP (Json News Transfer Protocol).

Here is its current "specification" (in French):
   http://www.nemoweb.net/?page_id=75

with even an RFC-like format:
   http://www.nemoweb.net/?p=1295

And, as John hinted at, people get then stuck in exchanges between an 
NNTP server and a captive webnews interface like:
   http://news.nemoweb.net/

Since 2014, no much move for JNTP, which is still used only by his creator.
Like SMTP and IMAP, NNTP operates in plain text.  A great advantage over 
other alternatives.

> carefully before standardizing yet another alternative way to do

+1

-- 
Julien ÉLIE

« – Vous ramassez des champignons sans les connaître ?
   – Et alors ? Ce n'est pas pour les manger mais pour les vendre. »

---

From: Rich Kulawiec <[email-masked]>
Date: Mon, 7 Nov 2016 17:47:41 -0500

1. I'll echo the comments about the very strong merits of a simple text
format.  It's served us well and withstood many transitions.  For something
more complex to be considered, it has to show (a) that it solves problems
that we concur are worth solving (b) that it doesn't create problems
we don't currently have (c) that it outperforms text *significantly*,
which to me means a minimum of an order of magnitude, preferably two.

2. I'll also echo the comments about whether or not JSON is worth dealing
with.  Very trendy, yes, but will it be so in 2019?  We *know* that
text will still be workable then.

3. If IMAP doesn't work as well as can be hoped for with mobile email
clients, then let's identify the specific issues and see if we can do
something about them in IMAP.

4. Using the web for email is a square peg-round hole problem that is
never going to admit elegant solutions.

---rsk

---

From: Keith Moore <[email-masked]>
Date: Mon, 7 Nov 2016 20:49:19 -0500

Entirely agree with John and Ned, and would go even further:

This project should absolutely not be taken on unless either
a) it's very narrowly scoped (much more narrow than the description 
seemed to imply) OR
b) it's such a good proposal that we'd feel comfortable deprecating one 
or more existing and widely-used standard email protocols

IMO the last thing we need is Yet Another Mail Access protocol that user 
agents need to support, and we might well be better off deprecating one 
of the ones we have.

I'm not a huge fan of IMAP and it's looking pretty baroque by now, and 
POP is so dysfunctional and ill-suited to mobile devices that phasing it 
out would make sense.   And a completely new protocol might actually be 
a good idea.   But *-over-HTTPS is almost certainly going to be worse 
than any of the above.

Keith

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 8 Nov 2016 08:54:59 +0000

Keith Moore writes:
> a) it's very narrowly scoped (much more narrow than the 
> description seemed to imply) OR
> b) it's such a good proposal that we'd feel comfortable 
> deprecating one or more existing and widely-used standard email 
> protocols

Both of the protocols I've read are wider scoped, and IMO better.

> IMO the last thing we need is Yet Another Mail Access protocol 
> that user agents need to support, and we might well be better 
> off deprecating one of the ones we have.

Stack Overflow is a nice indicator of developer interest and activity... as 
I read it, gmail's REST/JSON API has overtaken POP already and is 
overtaking IMAP for custom-written scripts. IMAP dominates for general 
MUAs.

POP looks like a good candidate for historic status.

> I'm not a huge fan of IMAP and it's looking pretty baroque by 
> now, and POP is so dysfunctional and ill-suited to mobile 
> devices that phasing it out would make sense.   And a completely 
> new protocol might actually be a good idea.   But *-over-HTTPS 
> is almost certainly going to be worse than any of the above.

Do you mean as a general statement, or are you saying that either 
fastmail's or gmail's REST/JSON APIs are worse than either IMAP or POP?

https://developers.google.com/gmail/api/guides/
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/

Arnt

---

From: Evert Mouw <[email-masked]>
Date: Tue, 8 Nov 2016 14:30:05 +0100

Op 8-11-2016 om 02:49 schreef Keith Moore:
> I'm not a huge fan of IMAP and it's looking pretty baroque by now, and POP is so dysfunctional and ill-suited to mobile devices that phasing it out would make sense.   And a completely new protocol might actually be a good idea.   But *-over-HTTPS is almost certainly going to be worse than any of the above.
Nobody likes current IMAP, but IMAP v5 never took off.

https://mailarchive.ietf.org/arch/search/?email_list=imap5

Also the Courier extension for sending email over IMAP also didn't gain much traction.

http://www.courier-mta.org/imap/INSTALL.html#imapsend

I still believe that a coordinated push to get a new version of IMAP would beat stuff such as JMAP.

Thanks,
Evert

---

From: Doug Royer <[email-masked]>
Date: Tue, 8 Nov 2016 10:15:31 -0700

> (2) Some of us believe, ... because they operate
> in plain text, ...

I thought JSON was 7-bit, and required encoding to use 8-bit data portable.

So, if I am correct, then JSON is not a gain when it comes to non-text data?

> (3) ...  Seems

The complexity of the requests and replies are the issue. If its mostly 
converting IMAP to JSON, the same problems are going to exist.

-- 

Doug Royer - (http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 07:15:41 -0800 (PST)

> I thought JSON was 7-bit, and required encoding to use 8-bit data portable.

JSON is almost always UTF-8. So no, it's unabashedly 8-bit. But so is 
the overwhelming major of the existing email infrastructure, so this is 
a wash.

> So, if I am correct, then JSON is not a gain when it comes to non-text data?

This is also a wash.

Binary MIME has been around for almost 25 years. SMTP was extended to
handle it not long after it was defined. And note that it's entirely
possible to use binary MIME only on SUBMIT and not depend on the
rest of the infrastructure handling it.

IMAP has had the ability to transfer parts in binary for a long time. And
IMAP also offers the ability to decode and transfer a encoded part on
the server side.

JSON doesn't support binary, but there are various extensions that do.

> converting IMAP to JSON, the same problems are going to exist.

Yes and no. Part of the problem with IMAP is that it was designed for
a very different sort of client than what we have now. (I note that the
problem SMTP is intended to solve hasn't changed nearly as much.) Getting
rid of historical baggage will definitely simplify things.

IMAP also uses an unnnecessarily large number of format variants.

OTOH, there's absolutely no doubt that there's considerable intrinisic
complexity in the message access space. Especially given the desire to
push more and more of the problem to the server.

On the flip side, basing things on HTTPS adds an enormous amount of inherent
complexity. But most of that complexity is hidden from the majority of
developers, who won't code HTTPS support directly, preferring instead to
use any one of the bazillion libraries out there that does it all for you.

My guestimate is that - and here I'm speaking only of the message access space
- that a HTTPS/JSON approach wins in terms of effective implementation
complexity, although it is far from clear to me by how much.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 08:13:29 -0800 (PST)

> https://mailarchive.ietf.org/arch/search/?email_list=imap5

An extensive rework the existing IMAP protocol is a daunting task. Not only
does it require in-depth knowledge of the existing protocol, it requires
considerable knowledge of how that protocol is used.

Like it or not, a key advantage of starting over is that you don't have to
worry about backwards compatibility with anything. You support the set of
features you deem important, and rely on various consistuencies to advocate for
the things they want.

> http://www.courier-mta.org/imap/INSTALL.html#imapsend

It's likely it gained no traction because it's poorly designed, to the point of
near-unusabiliy. In particular:

(1) The lack of immediate feedback that a message was actually sent is a
    deal-breaker for both interactive and non-interactive applications.
    SUBMIT servers do go offline from time to time.

(2) The suggestion that (1) can be solved by sending a copy to yourself
    is nothing short of absurd.

(3) The intrinsic incompatibility with how mail clients use Draft folders
    and Outbox is highly problematic.

I recall looking at this some time ago and thinking, "That's a support call
generator for sure." 

I note, however, that there have been any number of other specifications for
submit via IMAP, almost all of them better designed than this one but
neverthless all flawed in one way or another, and there has been essentially
zero uptake of any of them. What this says to me is that having two protocols
for access and submit much of a concern for developers as we'd like to think.

More generally, the ability to submit messages to the transport infrastructure
is almost certainly going to be desireable, but in and of itself it's not going
to be the thing that drives adoption.

> I still believe that a coordinated push to get a new version of IMAP would

For better or worse, proposals that rely on some sort of push happening almost
never succeed. And there's a push to develop an alternative to at least IMAP. A
push to revise and clean up IMAP, not so much.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 09 Nov 2016 11:42:31 -0500

Ned,

A few questions about some of your comments (again, just
questions);

--On Wednesday, November 09, 2016 07:15 -0800 Ned Freed
<[email-masked]> wrote:

>> The complexity of the requests and replies are the issue. If
>> its mostly converting IMAP to JSON, the same problems are
>> going to exist.

> Yes and no. Part of the problem with IMAP is that it was
> designed for a very different sort of client than what we have
> now.

Which design model are you talking about?   It seems to me that
the original specs (through and including RFC 1064 and IIR 1176)
had only what we now call online mode and was optimized for
"kiosk" uses.  With the exception of the "walk up to someone
else's shared machine" part, that seems very similar to what we
have today with the assumption that everything is on the server,
the client keeps messages only on a transient basis, and one
cannot access messages unless one is online.    It seems to me
that is not very different from where we are headed now, whether
via "dumb MUA" or "MUA implemented on a web interface".   It may
be more complex than needed because it contains some
optimizations are what, by today's standards, are low-bandwidth
environments but some of those features also have security
advantages (that we haven't talked about very much recently).

Now, when we added offline (simulated POP3 on steroids) and
disconnected modes with IMAP4, things got more complex.  From my
personal perspective, disconnected mode (more or less the
ancient PCMAIL model) is the only thing that makes sense for
people with poor or intermittent connectivity, especially if
they need to work offline sometimes and use multiple machines.
However, it has never been broadly and well-supported in MUAs
(there are, IMO, a few "good enough" MUA implementations but no
really good ones) so maybe the marketplace has told us there
aren't enough users to count.  

Now, I've never been a fan of the LISP-like command and response
syntax, but that is much more a matter of taste than
functionality.

> (I note that the problem SMTP is intended to solve hasn't
> changed nearly as much.)

I agree with you but have heard claims that the multiple relay
and alternate destination (mail exhannge) models have outlived
their usefulness in today's highly-connected Internet.  I don't
see things as that highly connected (even looking to the future,
SMTP would work much better in the presence of a DTN than, e.g.,
HTTP[S} and for much the same reason it worked well for delivery
servers that were only connected a few hours a week) and I think
the robustness those features enable as important.  However,
there is no disputing that mail transport could be considerably
simplified if we allowed only endpoint-to-endpoint connections
with no alternatives.  Such connections would also permit
straightforward feature negotiation.  That would be a big help
for, e.g., EAI/SMTPUTP8 and similar arrangements and might even
help blur the boundary between transport-level and end0t9-end
encryption.

>  Getting rid of historical baggage will definitely simplify
things.

But that is my question.  I don't see IMAP disconnected mode as
historical baggage.  We've ended up with some redundant ways to
do things as a result of newer and more powerful options and
commands and the old ways could be cleared out.   We could
probably make some different choices about data structures and
maybe even standardize some things that today just add to the
complexity of the options.  But I'm not sure how significant
those actually would be -- what are you thinking of?

> IMAP also uses an unnnecessarily large number of format

Yes.  See above.
 
> OTOH, there's absolutely no doubt that there's considerable
> intrinisic complexity in the message access space. Especially
> given the desire to push more and more of the problem to the
> server.

Exactly.
 
>,,,
> My guestimate is that - and here I'm speaking only of the
> message access space - that a HTTPS/JSON approach wins in
> terms of effective implementation complexity, although it is
> far from clear to me by how much.

But, unless we can get rid of IMAP (and some of the
functionality, like disconnected mode, that can't obviously be
supported over a purely HTTPS/JSON interface) entirely, it seems
to me that an HTTPS/JOSN approach is additive, requiring the
mail environment to support both it and IMAP for (at least) a
very long time.  So, while I agree with your guess, I don't see
the need to support an additional as a net gain, especially in
the short to medium term.

   john

---

From: Paul Smith <[email-masked]>
Date: Wed, 9 Nov 2016 17:01:35 +0000

> the short to medium term.

I'm looking at this 'JMAP' idea and wondering - why? What is it trying 
to solve that we can't already do.

'Making it more consistent' isn't a good enough reason IMHO. IMAP4 may 
be a bit weird, but it's here, and widely used. So, I'm not entirely 
sure what a 'JMAP' would achieve that we can't already do.

Is it so that people can write webmail apps in pure Javascript (without 
needing PHP/Python/whatever running on the server). That's a possible 
benefit of a HTTPS/JSON interface, but I'm not sure how important that 
would be long term.

Is it so that mail backends which don't really fit into the IMAP4 way of 
working (eg gmail) are easier to work with? That's a potential goal to 
make a whole new protocol, but it seems odd to try to standardise a 
whole new protocol just for that.

There may be something else; it may be that I'm underestimating the 
benefits I've seen. But, so far all I seem to see is 'people don't like 
IMAP4 so we want something new', which I'm not sure is a good enough reason.

So, what is 'JMAP' trying to give us that we can't already do? (Or that 
well designed IMAP4 extensions couldn't give us)

Will it be such an amazing advantage that developers (of both servers 
and clients) are willing to learn a whole new system and rewrite their 
software to deal with it? Will it let users do things they're desperate 
to do but can't do now, so they're clamouring at the doors of the 
developers to force them to do it? If not, then it'll be yet another 
protocol which may looks good but never gets widely implemented.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 9 Nov 2016 18:46:15 +0000

On 09/11/2016 16:13, Ned Freed wrote:

>> > I'm not a huge fan of IMAP and it's looking pretty baroque by now, 
>> and POP is so dysfunctional and ill-suited to mobile devices that 
>> phasing it out would make sense.   And a completely new protocol 
>> might actually be a good idea.   But *-over-HTTPS is almost certainly 
>> going to be worse than any of the above.

Indeed.

> features you deem important, and rely on various consistuencies to 

It is true. While I generally like IMAP model (surprise, surprise :-)), 
in some areas there are extensions on top of extensions, which are not 
as efficient or as elegant to implement as I would have liked.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 9 Nov 2016 19:03:36 +0000

Hi Ned,

If this work goes forward, there is definitely some wordsmithing to do 
to address your comments. Quick answers to some of them below:

> "Representation of email" would seem to imply that we're talking about 
No, the email format is unchanged.
> suite with the exception of SMTP.
Yes, at least this is the proposal.
> is, in standards terms at least, new.
This might be web push, but I agree that this is not clear.
> So we're clearly aiming to replace multiple protocols here.
Yes.
> the client with the server.
Yes, basically the IMAP side of things.
> Which would seem to exclude SUBMIT, unless
> you're doing SUBMIT with magic folders
Most likely, yes.
> and header fields or something
> similar.
>
No, see above.

---

From: Doug Royer <[email-masked]>
Date: Wed, 9 Nov 2016 12:56:26 -0700

On 11/09/2016 08:15 AM, Ned Freed wrote:
>
> ... so this is a wash. ...
> ... This is also a wash. ...

Good :-)

> handle it not long after itKaia FIT Boise was defined. And note that it's entirely

Yes, MIME is awesome. However it is now possible to send, transfer, and 
fetch full binary data. The encoding/decoding cycle now seems to be more 
of a habit, than a need. Time to reverse this process and only encode 
when requested?

> JSON doesn't support binary, but there are various extensions that do.

Yes, that is my point earlier.

>> ...The complexity of the requests and replies are the issue. If its mostly

Yes, it was mostly used by command line tools, that allowed you to save 
attachments as files and to decide what part of email you wanted to view.

> IMAP also uses an unnecessarily large number of format variants.

YES!

> OTOH, there's absolutely no doubt that there's considerable intrinsic

Problems that would need to be solved first:

What does a modern MUA's need?

I am not talking about what it takes to make an IMAP aware MUA work. I 
mean what functional needs do they have? (Fetch, fetch body part, 
notifications, get summary information, ...). If the goal is to make 
this new protocol so much like IMAP as to make the coding converting an 
IMAP MUA to a new-protocol faster? Or is the goal to do whatever the 
correct new thing is?

Most of the IMAP command were designed to make command line tools work. 
Those limitations no longer exist. You would get a summary of headers, 
pick which ones you wanted to read. Then perhaps download the attachment 
and save it to a file.

I think the new message fetch protocol could be much simpler.

Polling for email was needed years ago because most computers (or the 
programmers) had difficulty performing asynchronous notifications, or 
multi channel I/O.

Does the new protocol have the server parse the headers and hand them 
over in a more simple-client manner? (You have 10 headers, here they are 
(left side/right side). Or does it send a blob of headers and the MUA is 
responsible for the parsing? Same with body parts.

Does the new protocol help with securing the content in any way? If the 
can is opened, should this be in the that mix of what's changing? How 
can it not be a major issue?

What is needed at the functional level?

> complexity, although it is far from clear to me by how much.

I am not a HTTP(s) solves every protocol problem person.

If the WEB MUA had different needs, what exactly are they?
Do they want the HTML formatting done on the server side? Some of it? 
One of the many SMTP, IMAP, and POP ports are already opened on any 
firewall that needs email.

One problem with putting email on port 80/443 is that some companies 
need to control customer private data. And they would now have to block 
by content, not by port. Much is more difficult.

What functional differences are their between WEB email and application 
clients? What differences are their between mobile device needs and desktop?

Doing JSON because its the new thing as the reason, I would think is an 
insufficient reason to disrupt the email process.

This process is going to be a nightmare if the functional needs are not 
nailed down first.

--

---

From: Dave Crocker <[email-masked]>
Date: Thu, 10 Nov 2016 05:13:28 +0900

On 11/10/2016 1:13 AM, Ned Freed wrote:
> worry about backwards compatibility with anything.

The email technical community has, IMO, a really excellent track record 
of incremental enhancement.  Anytime someone calls for tossing out one 
email protocol or another (or all of them) and starting over, I press to 
first get a clear statement of what is needed that we don't have, a 
basis for claiminge community consensus for it, and the basis for 
believing we can't do yet-more incremental enhancement to get it.

IMAP seems to be the one case where starting over is strongly worth 
considering, and that is because of the continuing concern over its 
complexity and the established challenges of getting it to work well.

As with others, I'm always concerned about the hype of 
currently-fashionable technical choices; on the average, I take such 
hype as serving to argue /against/ whatever is being promoted, if only 
because the hype blinds folk from doing pragmatic analysis.  But JSON 
has been establishing a pretty reasonable track record of actual use in 
a variety of applications.

So let's look at the proposal in two simple terms:  Is there a need, and 
does the existing proposal look like a good foundation for satisfying it?

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Wed, 9 Nov 2016 20:34:01 +0000

One of the beauties I’ve always appreciated about IMAP was its conceptual model of how the message store and folders should work. Whether I agreed with all of the decisions on how somethings should work or not, the model was well thought out. The various warts^H^H^H^H^H/hacks^H^H^H^H^H^/enhancements we’ve added on top have mostly kept that model intact, and that’s a good thing. 

Any replacement needs to start with a similarly well-thought-out model. It needs to cover all of the cases of online use and offline caching with synchronization.

We’ve certainly learned a few things from the various IMAP extensions, and I agree that simplification can be achieved in various areas.

The least-interesting part, IMO, is the transport of commands, be it expressed in JSON, XML, or any other encoding-du-jour.

	Tony Hansen

On 11/9/16, 3:13 PM, "ietf-smtp on behalf of Dave Crocker" <[email-masked] on behalf of [email-masked]> wrote:

    

    
    d/
    
    -- 
    
       Dave Crocker
    
    _______________________________________________
    ietf-smtp mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 19:57:23 -0800 (PST)

> to address your comments. Quick answers to some of them below:

Sounds good.

> No, the email format is unchanged.

I note in passing that this is actually a departure from some of the
existing APIs, where messages are mapped into and out of JSON at least
some of the time.

> Most likely, yes.

And as I indicated previously, that's a problem, because magic folder are a
poor means of doing message submission, mostly because of their lousy error
handling.

I'll also note that none of the existing APIs I know of use magic folders. For
example, the Gmail API has two methods to send mail - message.send and
drafts.send, which return the error information you need. And the Outlook API
is very similar except that HTTP POST is used

But this really isn't the time to get into details. My point here is that if
you're going to do message submission, you need to get it right - something
essentially none of the current specifications do -  and that's going to
require an expansion of the model being proposed here.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 10 Nov 2016 09:28:50 -0500

> And as I indicated previously, that's a problem, because magic
> folder are a poor means of doing message submission, mostly
> because of their lousy error handling.

It is also worth noting that magic folders have been repeatedly
proposed for IMAP and rejected every time, always for reasons
that included the above.  If you decide to charter this, please
let's be sure that there are built-in mechanisms for adequate
review that considers those prior discussions.  In principle,
I'm not opposed to a carefully-considered decision that reaches
a different conclusion, but "carefully-considered" --and not
just by advocates of this work -- is key. 

>From my point of view, we went to the trouble to separate
message submission from SMTP for only two important reasons: to
make it clear what things a submission server could reasonably
do (and be expected to do) that an SMTP relay could not and to
permit (and encourage) a better error-handling model among
cooperating parties.  At least for me, that created a message
creation and submission model that was a partnership between the
creating MUA and a first-hop server, paralleling what we've
often called a "split MUA" model between an IMAP (or POP) server
and client at the receiving end.  Again, if a different model is
really needed --and, so far, I'm not convinced -- those
advocating it should be required to demonstrate to the community
that they understand all the implications of discarding or
bypassing the older one.

best,
     john

> require an expansion of the model being proposed here.

best,
   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 10 Nov 2016 14:43:38 +0000

> for access and submit much of a concern for developers as we'd 

Uhm. There's been zero uptake for stuff that requires clients to implement 
BOTH single-protocol AND two-protocol submission:

   if(unlikely condition)
       one implementation here
   else
       another that works in all cases

Why bother with the first branch if the second is unavoidable? Even if you 
like the first one better than the second? Even if the first is simpler?

Arnt

---

From: Dave Crocker <[email-masked]>
Date: Fri, 11 Nov 2016 04:58:55 +0900

(I hadn't seen the draft charter text until now, due to some 
unfortunately behavior in my email filters.)

As with others, I think the draft charter has a number of very basic 
problems.  I think there are several real issues and that some efforts 
to respond to them could be useful, but that we need far more clarity 
about needs and associated responses, and we need far more narrow 
sequencing of those efforts to respond.  In its current form, this draft 
charter is, IMO, far too scatter-shot and complex, and has far too 
little justifying foundation.

> environments they are operating in, particularly mobile and webmail.

As with others, I take 'representations' to mean formats.  And given 
what JSON is[0] that seems an appropriate interpretation even without 
that word.  So I take this asserted need as having a form of email data 
format that is an alternative to RFC 5322/MIME.

Prior to chartering, we should document just how extensive the current 
problem of multiple JSON email formats is, to establish the practical 
benefit of unifying it. The theoretical benefit should be obvious, but 
that's not enough to justify the cost of a working group.  We should 
establish real market need.

Having spent quite a bit of my early career on email gatewaying, I'll 
comment that getting a common interchange format is especially powerful. 
  The active protocols aren't irrelevant, but stabilizing the message 
object itself is, IMO, far more important. I'm more than a little biased 
about this strategic approach: It provided the unifying base for email, 
during the Internet's early growth into commercial markets.  And it 
happens that focusing on this narrow requirement permits end-to-end -- 
and I mean the real kind:  author to recipient, not just originating 
operator to receiving operator -- benefits without having to change the 
infrastructure, other than insertion of gateways.

Working groups need task serialization.  So I'll suggest that having an 
effort to standardize an JSON representation of an RFC5322/MIME object 
would be the highest-leverage first task.

And not a trivial effort...

As for the remaining effort(s)...

> channel with minimal round trips.

There is no justification given for this approach.  For each activity, 
there needs to be a clear and solid need documented, based on actual 
industry activities or at least industry statements.  Besides clarifying 
/what/ needs doing, it should serve to indicate likely industry uptake 
after the work is done.

> significant benefits in terms of battery life and network usage.

So, that sounds like some sort of need, but it isn't clear to me what it 
means.

> different protocols are authenticated separately

Ditto.  Worse, any claim that an alternative would be better needs to be 
justified both in terms of its considerable effort and its purported 
improvement.  Unification efforts have a habit of re-creating complexity 
and providing little actual benefit.

Note, for example, that the IETF email community debated whether 
submission should be independent of IMAP or integrated into it.  It was 
a one-year debate and, IMO, was conducted in a highly informed, diligent 
and constructive fashion.  That the result was to maintain separation 
should not be ignored...

>   supports stateless use, flood control and batched operations.

Since client/server email access typically benefits from having the 
server retain state about the client's activities, I can't tell whether 
this actually means stateless lower-level transport or whether it really 
does mean a stateless email protocol.  So this needs to be made much 
more clear, as to the what and the why, as well as to the benefit that 
will accrue.

>   with the data structures expressed as JSON.

Given the considerable complexity of HTTP, I continue to fail to see why 
making it be a universal, lower-layer transport is so popular.  Again, 
the need and the benefit need to be documented.

>   developers of JMAP servers to ensure they conform to the

This last is useful, but I think it's not typically an IETF work product?

A charter is more than a work statement.  It is a marketing tool for 
recruiting industry support.  Hence the need for explanatory text that 
makes needs and benefits clear, as well as specification of deliverables.

d/

[0] http://www.json.org/

-- 

   Dave Crocker

---

From: Ned Freed <[email-masked]>
Date: Fri, 11 Nov 2016 07:26:10 -0800 (PST)

> like the first one better than the second? Even if the first is simpler?

The obvious counterexample here is SMTP pipelining, which is widely implemented
and used. (According to my logs, ~500 out of ~800 connections used pipelining
when talking to my server yesterday.)

But more interesting counterexamples include, but are probably not limited to,
binary SUBMIT and BURL. Usage of them is low but IME definitely not zero. (And
before you ask, the answer is no, I can't provide any specific details on which
of our customers use them. But given their nature you can probably figure out
what sort of client might find them useful.)

Really, it's all about *operational* cost versus benefit.  If the operators saw
significant operational benefit in a single protocol solution, *or* saw
significant operational cost associated with using two protocols that a single
protocol solution would eliminate, they would be telling their client and
server implementors to code it up.

But AFAIK we have yet to receive a single request for such support. Which says
to me that neither condition is being met.

				Ned

---

From: Hector Santos <[email-masked]>
Date: Mon, 14 Nov 2016 13:19:24 -0500

On 11/8/2016 3:54 AM, Arnt Gulbrandsen wrote:
>> b) it's such a good proposal that we'd feel comfortable deprecating
>> one or more existing and widely-used standard email protocols
>
>> IMO the last thing we need is Yet Another Mail Access protocol that
>> user agents need to support, and we might well be better off
>> deprecating one of the ones we have.
>
> Stack Overflow is a nice indicator of developer interest and
> activity... as I read it, gmail's REST/JSON API has overtaken POP
> already and is overtaking IMAP for custom-written scripts. IMAP
> dominates for general MUAs.
>

Of course, exploring alternatives is welcomed, but please don't 
advocate this.  For decades, millions, if not billions, people, 
servers, devices, businesses and market rely on have a basic POP3 
protocol. Its a fast, simple entry protocol. All else cost money. POP3 
works just fine and even better these days with improves speeds and 
network reliability.

Thanks.

-- 
HLS

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Nov 2016 10:27:32 -0800 (PST)

Apologies for the late response on this one.

> advantages (that we haven't talked about very much recently).

Pretty much the entire design model is canted away from what I see being needed
by modern clients.

One assumption the old model makes is that bandwidth is cheap enough that
there's no need to highly optimize metadata transfer. And to some extent this
also applies to message data, e.g., the lack of support for message summaries
and thumbnails, but that's a more complex topic.

This matters because with mobile client view timliness is critical, but not at
the expense of bandwidth. And push notifications only get you half of the way
there - once notified you have to connect and update.

Sit down with a mobile client designer sometime, and you'll quickly see that in
order to build the sorts of message displays they want without transferring far
more data than they need or can afford, you need to engage a large number of
IMAP extensions, and even then you're unlikely to achieve the exact result they
would prefer.

And then there's search. A powerful search capability is now essential, but
IMAP search assumes a particular set of semantics that are simultaneously not
what a mobile client needs (nobody is going to spend time entering complex
search terms on their phone) as well as being misaligned with modern text
search engines, making it difficult to implement on the back end. (What
typically happens is implementations vary substantially from what the
specifications call for, and vary in different ways.)

Then there's the stuff that nobody cares about that you have to support, like
subcriptions.

I could go on and list more stuff, but I think this makes the point that
there's a mismatch. And really, given how the landscape has changed and how old
some of these specifications are, I see no reason for surprise.

> aren't enough users to count.

Offline support is another thing that's increasingly superfluous now. Mobile
devices are expected to always be online and to present an up-to-date view of
the world at all times.

Of course the sync capbilities designed to support offline mode can also be
used to minimize data transfer for a connected client, but even so, it's not
quite the same problem.

> syntax, but that is much more a matter of taste than

Unfortunately, syntax matters more than it really should. I previously pointed
out that the complexity of HTTPS in this context is irrelevant because "there's
a library for that". The same is true for JSON - the automapping mapping of
JSON to and from data structures that many languages offer is very convenient.

> their usefulness in today's highly-connected Internet.

Even if it's true, this is all external to SMTP itself, and I don't think
it's relevant here.

> I don't
> with no alternatives.

Uh, no, not really. To the extent that implementation of message transfer is a
complex problem, it's mostly in the splitting of messages and the handling of
errors. These problems don't go away when you limit things to one hop. In fact
one of the nice things about SMTP is that you don't know and don't need to know
how many hops there are ahead of you.

The complexity of transitions through multiple ADMDs grots up the security and
spam situation, but those are for the most part externalities to the SMTP
implementation.

Additionally, this notion of getting rid of multiple hops across the Internet
ignores the present day reality that as the size of a "typical" mail system
deployment has increased, so has the number of hops messages often take inside
the deployment.

> Such connections would also permit

Perhaps, but again, I don't see the relevance to the matter at hand.

> >  Getting rid of historical baggage will definitely simplify things.

> those actually would be -- what are you thinking of?

IMAP disconnected mode support isn't baggage because it provides capabilities
that are useful to mobile connected clients, albeit in a somewhat suboptimal
fashion.

That said, there absolutely is substantial baggage in IMAP.

> the short to medium term.

Just a guess on my part, but I rather suspect that disconnected mode is
easier to do on top of a protocol designed to optimize mobile sync than
the other way around.

Of course this ignores the cost of conversion. But I don't think you can
necessarily make the case based on protocol features alone.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 06:49:10 +0900

I agree with almost everything you said here, Ned, except for the idea that
offline isn't important. For those of us who travel internationally, being
able to look up travel info on your phone before you have your
international SIM is crucial, and this is just one example of a situation
where this is important.

---

From: Doug Royer <[email-masked]>
Date: Mon, 14 Nov 2016 17:23:48 -0700

> (lots of stuff ...)

Ned: YES - YES.

I have felt that way for years.

In response to to Ted's concerns about off-line.

I use SeaMonkey (a Thunderbird spin) for my email. When I setup a mail 
account with it, one of the options is for it to cache the email account 
for offline usage.

I would think this is mostly a client issue. In a future protocol, when 
checking for new message data, pull all of the summary data, and 
optionally all of the body parts. Its a client issue. Save the data, or 
not. If you pull it all, then the client is free to save it, or not.

--

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 09:30:14 +0900

Making offline support a "client issue" means that every client will
do it differently, and there will be no hooks in the protocol to make
it easy/consistent.   Why _wouldn't_ you want to document offline
support?

On Tue, Nov 15, 2016 at 9:23 AM, Doug Royer <[email-masked]> wrote:
> account with it, one of the options is for it to cache the email account for
> offline usage.
>
> checking for new message data, pull all of the summary data, and optionally
> all of the body parts. Its a client issue. Save the data, or not. If you
> pull it all, then the client is free to save it, or not.
>
>
> --
>

---

From: Ned Freed <[email-masked]>
Date: Mon, 14 Nov 2016 22:38:53 -0800 (PST)

> international SIM is crucial, and this is just one example of a situation

There's a difference between being able to operate offline and a protocol
specifically designed to support offline operation.

As you note, the former is still useful, but is, as Doug notes, mostly a matter
of client support. (Unfortunately, I suspect that as a practical matter you're
going to have ever-increasing difficulty getting client developers to
prioritize it.)

To the extent it's a protocol issue, it's mostly a subset of the mobile
sync problem.

Or, to put it another way, it seems the mobile sync problem has more
requirements and is more constrained than the offline sync problem. And for
better or worse, the design focus of IMAP was and is on the offline sync
problem.

All that said, it's not entirely clear to me that the design focus of the
current crop of JMAP proposals is geared towards the mobile client sync problem
either. At least some of what I've seen seems more oriented towards support of
business workflows, which is a very different (and IMO much simpler) problem.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Tue, 15 Nov 2016 16:30:14 +0900

Yup.   The reason I keyed in on what you said is that I've had the
same thought about IMAP for a long time--I would like to have a much
less chatty sync process.

---

From: Tony Finch <[email-masked]>
Date: Tue, 15 Nov 2016 12:10:19 +0000

Is everyone here using the term "offline mode" to mean the same thing?

RFC 1733 defines "offline mode" to mean POP3-style delete-from-server.
"Disconnected mode" is the term for a client that has a local cache
or mirror of what is on the server.

Approxomately no-one wants RFC 1733 offline mode. (Bah, the number of
times I have had to recover mail for someone who accidentally pointed an
offline-mode client at their account.)

How much do the necessary protocol features differ between a full-fat
disconnected mode client (a mirror on the client of the entire
account on the server) vs a poorly-connected mobile client which needs to
sync a smaller window on the account? Concurrent access to multiple
mailboxes? what else?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Trafalgar: Easterly or northeasterly 5 or 6, occasionally 4 at first, but 6 to
gale 8 in southeast. Moderate or rough. Fair. Good.

---

From: Doug Royer <[email-masked]>
Date: Tue, 15 Nov 2016 12:53:06 -0700

Documentation:

	Keep copies of the email if you want.

	Outbound is easy, queue it until you connect, then send.

	If the server has emails deleted, that the client still has,
	delete them from your client if you want.

Documentation done? :-)

I do not understand why its a protocol issue. What is needed? What would 
make it easy/consistent?

--

---

From: Hector Santos <[email-masked]>
Date: Tue, 15 Nov 2016 18:04:45 -0500

> or mirror of what is on the server.

Having had one of the earliest offline mail systems in the market 
place (even before POP3/IMAP/SMTP with QWK/OPX mail systems which 
still exist today), deleting backend mail from server was never was 
user right. It was more appropriate to save/record/remember "last 
read/download" positions in a mail database per user.

> offline-mode client at their account.)

Right, "Roaming Users" allowed you to "re-download" the mail but it 
depended on an option such as "[X] Mark downloaded mail as Received."

If checked, Roaming users will not get the mail again as they went 
from device to device, i.e. Home vs Office.  If Off, each new device 
setup would get the mail again.  This could put a strain on systems 
with automated mail maintenance, packing, archiving of "already 
received" mail.

> sync a smaller window on the account? Concurrent access to multiple

A mail "exchange" protocol should work both ONLINE and OFFLINE.  The 
Offline mode should offer a near, if not full 100%, Emulation and 
Experience of whats possible online without violating the security, 
i.e. whats not possible online, MUST not be possible offline.

-- 
HLS

---

From: Ted Lemon <[email-masked]>
Date: Wed, 16 Nov 2016 08:10:08 +0900

Good clarification--thanks, Tony!

On Wed, Nov 16, 2016 at 8:04 AM, Hector Santos <[email-masked]> wrote:
> Having had one of the earliest offline mail systems in the market place
> (even before POP3/IMAP/SMTP with QWK/OPX mail systems which still exist
> today), deleting backend mail from server was never was user right. It was
> more appropriate to save/record/remember "last read/download" positions in a
> mail database per user.
>
> Right, "Roaming Users" allowed you to "re-download" the mail but it depended
> on an option such as "[X] Mark downloaded mail as Received."
>
> If checked, Roaming users will not get the mail again as they went from
> device to device, i.e. Home vs Office.  If Off, each new device setup would
> get the mail again.  This could put a strain on systems with automated mail
> maintenance, packing, archiving of "already received" mail.
>
> A mail "exchange" protocol should work both ONLINE and OFFLINE.  The Offline
> mode should offer a near, if not full 100%, Emulation and Experience of
> whats possible online without violating the security, i.e. whats not
> possible online, MUST not be possible offline.
>
> --
> HLS
>
>
>

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 16 Nov 2016 10:27:15 +0900

Hi,

I've just asked IETF Secretariat to create a new mailing list: [email-masked]. Please subscribe if you are interested (or even curious) about JMAP. You can visit
 https://www.ietf.org/mailman/listinfo/jmap
or send "subscribe" email to [email-masked].

Can I please ask further discussions of JMAP and JMAP Chartering to be done there.

Thank you,
Alexey

---

From: Doug Royer <[email-masked]>
Date: Tue, 15 Nov 2016 21:18:08 -0700

I have started replying to  [email-masked] only for the JMAP issues.

--

---

From: Jacob Palme <[email-masked]>
Date: Thu, 17 Nov 2016 18:47:07 +0100

The MHTML format may be of interest as input to this group. It compiles encapsulation of various parts in MIME-formatted e-mail.

See:
https://en.wikipedia.org/wiki/MHTML
https://tools.ietf.org/html/rfc2557
— 
Jacob Palme professor emeritus Stockholm University
for more info see https://people.dsv.su.se/~jpalme/

> On 07 Nov 2016, at 19:05, Ned Freed <[email-masked]> wrote:
>

---

From: Brandon Long <[email-masked]>
Date: Mon, 21 Nov 2016 21:38:11 -0800

I've said this to some client developers directly, but it's good to share
the concept more widely:

There should be no "full mirror" clients, only disconnected state clients
which make different choices of window size.  There are Gmail accounts
today which would exceed the ssd in many laptops, or the monthly bandwidth
allotment of some broadband customers.  A full sync is useful for backups,
but a regular client would be wise to make full sync a special case of
"reasonable cache space on this device/connection happens to be larger than
the user's account".

Granted, I would also argue against connected clients in general, since the
next billion users typically have poor/expensive connections, and many
users are on mobile networks where the quality can vary greatly from block
to block.  Network usage is also more likely to be a WAN these days, with
the higher latency that implies.  Also, storage size is mostly to the point
where a local cache can be fairly large compared to most content... OTOH,
shared computers may make local sync or install less desirable, but I think
most of those cases tend to web clients these days.

Brandon