Subject: Re: [MORG] WGLC on draft-ietf-morg-inthread-00

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 14:12:44 +0100

Cyrus Daboo writes:
> Hi Timo,
>
> --On February 13, 2009 5:50:53 PM -0500 Timo Sirainen <[email-masked]> wrote:
>
>> The IMAP SEARCH=INTHREAD and THREAD=REFS Extensions
>>
>> <http://www.ietf.org/internet-drafts/draft-ietf-morg-inthread-00.txt>
>
> Technically OK, but I was left wondering why the heck I would want to 
> implement the new SEARCH capabilities if I were a client.

This document is meant to answer some questions. It doesn't tell you to 
ask those questions.

In my not at all humble opinion, there's a four-step process:

  1. You need to want to do something.
  2. You look for ways to do it.
  3. You find some RFC.
  4. You read it.

4 comes after 1. Telling people in step 4 that they should want 
something is no good.

This implies that an RFC must be findable for the people whose questions 
it answers, then it must answer their questions (and do so clearly 
enough that they understand it).

I added some text to the address-search document, answering the all too 
common question "how do I implement so-and-so if the server does and 
doesn't implement this extension"? I guess similar stuff would be 
useful here.

Also, I'll dream up some google searches and see that the draft is found.

> I think the document needs to have more examples or concrete use cases 
> to better explain why these changes are important and how clients 
> would make use of them, and what the benefit to the server is to 
> implement.

What are some RFCs that do this and benefit from it?

> For this document an explanation of what clients are doing today with 
> threads and IMAP protocol use with that vs what would happen with the 
> new extension would go a long way to justifying implementing this.

I'll add that. (Where will you implement it?)

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 26 Mar 2009 08:25:50 -0700 (PDT)

On Thu, 26 Mar 2009, Arnt Gulbrandsen wrote:
> This document is meant to answer some questions. It doesn't tell you to ask 
> those questions.

It also begs the question: why the heck I would want to implement any of 
this if I were a server.

What is the business case to expend the substantial resources (real $$$) 
to implement these extensions?  Where are the customers who want these 
extensions?  How many additional sales will be made if these extensions 
are implemented?

> 2. You look for ways to do it.

In other words, this working group is producing a solution, in hope that 
eventually there will be a problem for that solution to solve.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 16:26:00 +0100

Alexey Melnikov writes:
> Most of my comments are editorial:

Done.

> I think the document needs to define what "normalized" means here.

OK.

>> 4.  The THREAD=REFS Thread Algorithm
>>
>>    The THREAD=REFS thread algorithm is defined as the part of
>>    THREAD=REFERENCES (see [RFC5256]) which concerns itself with the
>>    References, In-Reply-To and Message-ID fields.  THREAD=REFS ignores
>>    Subject.
>
> Does this mean that the algorithm is effectively ignoring step (5) 
> from RFC 5256?

Yes.

>>    It is explicitly permitted for the server to persistently store
>>    threading information, even if this causes the server to return
>>    different information than it would otherwise. This can happen if
>>    the first messages in a thread are deleted, for example.
>
> This is a strange paragraph. Why is it needed?

If you thread incrementally as messages are added to the mailbox, and 
some messages are deleted, then your results may be slightly different 
from if you thread at read time. I want that to be OK.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 16:32:38 +0100

Mark Crispin writes:
>>> Technically OK, but I was left wondering why the heck I would want 
>>> to implement the new SEARCH capabilities if I were a client.
>> This document is meant to answer some questions. It doesn't tell you 
>> to ask those questions.
>
> It also begs the question: why the heck I would want to implement any 
> of this if I were a server.
>
> What is the business case to expend the substantial resources (real 
> $$$) to implement these extensions?  Where are the customers who want 
> these extensions?  How many additional sales will be made if these 
> extensions are implemented?

Are you seriously suggesting that any of this should go into an RFC?

> In other words, this working group is producing a solution, in hope 
> that eventually there will be a problem for that solution to solve.

That risk certainly exists. Each of us has to decide whether this is 
worth the effort.

I'm not going to try and persuade you that it's worth anything _for_you_.

(FWIW, in my case the capabilities are valuable but providing an IMAP 
interface is of dubious value.)

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 26 Mar 2009 09:01:04 -0700 (PDT)

> Are you seriously suggesting that any of this should go into an RFC?

I am saying that an RFC should have a compelling reason to exist.  In 
those cases where this is not immediately obvious, it needs to present the 
justifications for its existence.

-- Mark --

http://panda.com/mrc

---

From: Barry Leiba <[email-masked]>
Date: Thu, 26 Mar 2009 12:17:22 -0400

FWIW, I agree with Mark.  Writing something up just because it sounds
like a good idea is fine, as far as it goes, but we should not
underestimate the real cost of putting something into the standards
track.  If we all say, "Yeah, that's cool," but then no one implements
it in real products[1], we've wasted a lot.

Barry

[1] Well, no one except the author and his mom, or whatever; you know
what I mean.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 17:34:44 +0100

> it in real products[1], we've wasted a lot.

I don't mind dropping either just S=I or the entire document, if that's 
WG consensus. Timo/Randy, will you take a hum tomorrow morning?

Personally I think that most of the MORG documents are well-justified 
and sensible. Rather more so than some other recent IMAP-related 
documents. But that's just my opinion. If I'm an outlier, that's okay.

> [1] Well, no one except the author and his mom, or whatever; you know 

Sure.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 26 Mar 2009 09:39:06 -0700 (PDT)

On Thu, 26 Mar 2009, Arnt Gulbrandsen wrote:
> Personally I think that most of the MORG documents are well-justified and 
> sensible. Rather more so than some other recent IMAP-related documents. But 
> that's just my opinion. If I'm an outlier, that's okay.

For what it is worth, I share your opinion of the other recent 
IMAP-related documents.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 17:41:32 +0100

> For what it is worth, I share your opinion of the other recent 

I said "some", you didn't ;) I wonder whether we'd write the same list.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 26 Mar 2009 17:50:37 +0100

Alexey Melnikov writes:
> Some additional comments:
>>    The INTHREAD search-key matches a message if its subsidiary 
>>    search-key matches at least one message in the same thread as the 
>>    message.
>
> Every time I looked at this sentence it makes me 20 seconds in order 
> to understand it.
> Wouldn't it be better to say that first all messages matching the 
> subsidiary search-key are found. Then all messages belonging to the 
> same threads are returned.

Like this perhaps? "The INTHREAD search-key takes another search-key as 
argument. If the argument matches any message in a thread, then 
INTHREAD matches all messages in that thread."

>> 3.5. The THREAD=* Search Return Option(s)
>>
>>    The THREAD=* search return options enables the client to select 
>>    which threading algorithm the server uses when processing 
>>    INTHREAD, THREADROOT and THREADLEAF as part of a SEARCH command. 
>>    If THREAD=* isn't specified, then the default for the SEARCH 
>>    command is THREAD=REFS.
>
> I hate to raise this issue, but should we establish an IANA registry 
> for search return options?

I see your point, but I'd like to see a little more ESEARCH use in 
clients before bothering with that.

Arnt

---

From: Mark Crispin <[email-masked]>
Date: Thu, 26 Mar 2009 10:10:48 -0700 (PDT)

On Thu, 26 Mar 2009, Arnt Gulbrandsen wrote:
>> For what it is worth, I share your opinion of the other recent IMAP-related 
>> documents.

With the notable exception of the security extensions (STARTTLS, 
LOGINDISABLED, AUTH=) pretty much all the IMAP extensions, past and 
present, have no market value to any vendor determining a feature set for 
a new server.

A few older extensions have use cases for a single client, and thus are 
worthile only if you have many customers who use that client.

Some others were good ideas but were broken by design and/or were never 
used as intended.  The subscription stuff in the base specification also 
falls into this category.

The less said about the remainder, the better.

-- Mark --

http://panda.com/mrc

---

From: Timo Sirainen <[email-masked]>
Date: Thu, 26 Mar 2009 13:41:47 -0400

On Mar 26, 2009, at 11:32 AM, Arnt Gulbrandsen wrote:

>> It also begs the question: why the heck I would want to implement  
>> any of this if I were a server.
>>
>> $$$) to implement these extensions?  Where are the customers who  
>> want these extensions?  How many additional sales will be made if  
>> these extensions are implemented?
..
> (FWIW, in my case the capabilities are valuable but providing an  
> IMAP interface is of dubious value.)

In my case it's pretty much the same. The functionality is required to  
implement features that customers want (like "gmail-like conversation  
views"), but it's mainly used in configuration files. The IMAP part  
just came without any extra code, so it would be nice to get it  
standardized. If INTHREAD doesn't become a standard, Dovecot will  
still have a X-INTHREAD search key.

But I do find THREAD=REFS very useful as an alternative to  
THREAD=REFERENCES.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 27 Mar 2009 05:22:48 -0700

Arnt Gulbrandsen wrote:

> Like this perhaps? "The INTHREAD search-key takes another search-key 
> as argument. If the argument matches any message in a thread, then 

Yes, this is much better.

> clients before bothering with that.

Ok.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 27 Mar 2009 05:30:10 -0700

Arnt Gulbrandsen wrote:

> This implies that an RFC must be findable for the people whose 
> questions it answers, then it must answer their questions (and do so 
> clearly enough that they understand it).

Last sentence: I agree.

> I added some text to the address-search document, answering the all 
> too common question "how do I implement so-and-so if the server does 
> and doesn't implement this extension"? I guess similar stuff would be 
> useful here.

Sure.

>> I think the document needs to have more examples or concrete use 
>> cases to better explain why these changes are important and how 
>> clients would make use of them, and what the benefit to the server is 
>> to implement.
>

I can't remember any RFC doing this from the top of my head, but I think 
I've done something like this in SEARCHRES document.

I actually agree with this comment from Cyrus and I am happy to suggest 
some text.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 27 Mar 2009 15:35:56 +0100

Alexey Melnikov writes:
> I actually agree with this comment from Cyrus and I am happy to 
> suggest some text.

Great.

Something about that comment reallly pushed my buttons. Sorry about 
that. I know I'm a difficult kind of person. Anyway, I think suggested 
text will please me much more.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 27 Mar 2009 19:33:59 +0100

> Yes, this is much better.

I revised both drafts based on room discussion already. I don't think I 
can get them out until Monday though, no time for nits now. But 
everyone's in a post-coital vacuum for a few days anyway, right?

Arnt

---

From: Curtis King <[email-masked]>
Date: Fri, 27 Mar 2009 12:30:51 -0700

On 26-Mar-09, at 10:10 AM, Mark Crispin wrote:

> present, have no market value to any vendor determining a feature  
> set for a new server.

I did a check on my server to see what extensions are being used by  
clients and there only seems to-be 10 extensions in use by the most  
common imap clients.

Clients: Apple Mail, Thunderbird, Mobile Mail, Outlook and php imap  
client. Mark your client is missing but I'm sure you can add any  
missing extensions to the list :-)

Extensions: STARTTLS, AUTH=, LOGINDISABLED, UIDPLUS, LITERAL+,  
NAMESPACE, IDLE, ACL, UNSELECT and QUOTA.

I'm sure there are some more clients which use a few more extensions,  
but that still means ~16 extensions we (Isode) have implemented are  
not being used. Not a very good ratio. Yet, here we are planning on  
adding more.

ck

---

From: Mark Crispin <[email-masked]>
Date: Fri, 27 Mar 2009 13:24:54 -0700 (PDT)

On Fri, 27 Mar 2009, Curtis King wrote:
> I did a check on my server to see what extensions are being used by clients 
> and there only seems to-be 10 extensions in use by the most common imap 
> clients.

I am not particularly surprised.

> Clients: Apple Mail, Thunderbird, Mobile Mail, Outlook and php imap client. 
> Mark your client is missing but I'm sure you can add any missing extensions 
> to the list :-)

I think that Pine/Alpine remains a fairly important client!  ;-)

I'm surprised by your report about PHP-IMAP, since it uses the same 
internal engine used by Pine, Alpine, UW imapd, and Panda imapd.  Maybe 
PHP-IMAP doesn't offer a sort capability.

So, with that said...

> Extensions: STARTTLS, AUTH=, LOGINDISABLED, UIDPLUS, LITERAL+, NAMESPACE, 
> IDLE, ACL, UNSELECT and QUOTA.

Of the listed extensions:

Pine/Alpine uses STARTTLS, AUTH=, LOGINDISABLED, NAMESPACE, and UNSELECT.

Pine/Alpine does not use UIDPLUS, LITERAL+, or IDLE, but these are all 
implemented in UW/Panda imapd.  I do not dispute their value, other than 
disputing the claim that LITERAL+ is a satisfactory alternative to 
MULTIAPPEND.

Pine/Alpine does not use ACL and QUOTA, and these are not implemented in 
UW/Panda imapd.  I continue to believe that both extensions are broken by 
design, but I don't care to continue that debate.

In addition to the above named extensions:

Pine/Alpine uses SCAN and THREAD=.  A major advertised feature depends 
upon these.  Performance is miserable without it.

Pine/Alpine uses MULTIAPPEND for cross-mailstore copies, and performance 
is miserable without it.  LITERAL+ does not address the underlying cause 
of the problem.  Nonetheless, the biggest use of MULTIAPPEND is in 
external administrative programs to move users between servers; I can't 
imagine IMAP being used for that purpose without MULTIAPPEND.

Pine/Alpine also use LOGIN-REFERRALS and MAILBOX-REFERRALS, but I doubt 
that any tears would be shed by their demise.  AFAIK, Cyrus was the only 
server that implemented it.

I implemented several other extensions in UW/Panda imapd mostly because 
they were trivial, low-hanging fruit; but I have no illusions that any of 
these are of any real value.

> I'm sure there are some more clients which use a few more extensions, but 
> that still means ~16 extensions we (Isode) have implemented are not being 
> used. Not a very good ratio. Yet, here we are planning on adding more.

I agree.  I have largely kept quiet here because I think that the entire 
exercise is more or less harmless, albeit silly.  As I was responsible for 
similiar silliness in my youth (my very first RFC was the TELNET Logout 
option), indulgent toleration seems to be the best policy.

-- Mark --

http://panda.com/mrc

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 27 Mar 2009 21:41:14 +0100

Mark Crispin writes:
> internal engine used by Pine, Alpine, UW imapd, and Panda imapd.  
> Maybe PHP-IMAP doesn't offer a sort capability.

PHP offers c-client, but AFAICT the popular PHP webmail programs don't 
use it. I looked at squirrelmail and roundcube now since those are 
often mentioned and I happened to have the source handy: SM uses some 
homebrew IMAP code, Roundcube uses something called 'ilohamail imap 
library'.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 27 Mar 2009 21:42:08 +0100

> not being used. Not a very good ratio. Yet, here we are planning on 

A bit of discussion of the reasons seems like a good idea. I have 
opinions but it's late and I'm too tired now. I'll try to post 
something during the coming week.

Arnt

---

From: Joseph Yee <[email-masked]>
Date: Fri, 27 Mar 2009 17:05:19 -0400

Horde IMP uses PHP-IMAP, which is c-client based.

ilohamail is another open source mail client, AFAIK, it was not active  
for awhile.  Roundcube uses it IMAP code base.

Regards,
Joseph

On 27-Mar-09, at 4:41 PM, Arnt Gulbrandsen wrote:

> PHP offers c-client, but AFAICT the popular PHP webmail programs  
> don't use it. I looked at squirrelmail and roundcube now since those  
> are often mentioned and I happened to have the source handy: SM uses  
> some homebrew IMAP code, Roundcube uses something called 'ilohamail  
> imap library'.
>
> Arnt
> _______________________________________________
> MORG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/morg
> Note Well: http://www.ietf.org/maillist.html

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 28 Mar 2009 08:47:52 +0100

>  for awhile.  Roundcube uses it IMAP code base.

I'm sure I've also seem something else that uses php-imap/c-client a 
while ago. Can't recall the name, but it was a four-letter abbrevation.

It doesn't matter. All I wanted to do was explain the discrepancy betwen 
what c-client does and Curtus sees that PHP webmail clients do.

Arnt

---

From: Curtis King <[email-masked]>
Date: Sat, 28 Mar 2009 11:13:40 -0700

On 27-Mar-09, at 2:05 PM, Joseph Yee wrote:

> ilohamail is another open source mail client, AFAIK, it was not  
> active for awhile.  Roundcube uses it IMAP code base.

Roundcube is the webmail I use, so that's why SORT or THREAD are  
missing from my list  of extensions.

ck