Subject: [Extra] draft-ietf-extra-imap4rev2-04 review

From: Timo Sirainen <[email-masked]>
Date: Mon, 25 Mar 2019 14:04:58 +0200

Hi,

Here's my review of draft-ietf-extra-imap4rev2-04:

> 1.2.  Conventions Used in This Document
> 
>    "Session" refers to the sequence of client/server interaction from
>    the time that a mailbox is selected (SELECT or EXAMINE command) until
>    the time that selection ends (SELECT or EXAMINE of another mailbox,
>    CLOSE command, or connection termination).

UNSELECT missing from the list

> 2.3.1.1.  Unique Identifier (UID) Message Attribute
>    The unique identifier validity value is sent in a UIDVALIDITY
>    response code in an OK untagged response at mailbox selection time.
>    If unique identifiers from an earlier session fail to persist in this
>    session, the unique identifier validity value MUST be greater than
>    the one used in the earlier session.

I see this was already in RFC 3501, but this "MUST be greater" has commonly been violated by servers. I think it should say "MUST be different". This is mainly because RENAME commonly doesn't change UIDVALIDITY, so:

RENAME mailbox-with-uidvalidity2 mailbox-with-uidvalidity2b
RENAME mailbox-with-uidvalidity1 mailbox-with-uidvalidity2

Would result in mailbox-with-uidvalidity2 shrinking UIDVALIDITY from 2 to 1.

> 3.4.  Logout State

>    A server MUST NOT unilaterally close the connection without sending
>    an untagged BYE response that contains the reason for having done so.

Also already in RFC 3501, but this MUST NOT has to be violated sometimes. For example otherwise it would allow client to FETCH a large message and keep reading it very very slowly over days, and server wouldn't be allowed to disconnect it, because it can't send BYE in the middle of the large mail literal.

> 4.5.  NIL

This is pretty easy to get wrong. It would help to have some examples of NIL and "NIL" in astring and nstring and what they mean. Like:

* LIST () "/" NIL
is equivalent to:
* LIST () "/" "NIL"

but

* FETCH 1 (BODY[1] NIL)
is not equivalent to:
* FETCH 1 (BODY[1] "NIL")

> 5.1.  Mailbox Naming

> servers MAY accept a de-normalized UTF-8 mailbox name and convert it to Net-Unicode prior to mailbox creation).
and:
> Some server implementations
>    are fully case-sensitive in ASCII range; others preserve case of a
>    newly-created name but otherwise are case-insensitive; and yet others
>    coerce names to a particular case.

This seems rather difficult from client's point of view. It CREATEs a mailbox, but the client can't find it anymore because it changed. Seems to me it would be less strange if the server simply failed the CREATE entirely.

If this is allowed for CREATE, is such conversion done for SELECT/EXAMINE/APPEND as well? What about LIST "" <non-normalized name> - would it return the normalized name?

> A.1.  Mailbox International Naming Convention
> 
>    By convention, international mailbox names in IMAP4rev2 are specified
>    using a modified version of the UTF-7 encoding described in [UTF-7].

This chapter needs removal or rewrite.

> 6.2.2.  AUTHENTICATE Command
> 

> Client and server
>       implementations SHOULD implement additional [SASL] mechanisms that
>       do not use plaintext passwords, such the GSSAPI mechanism
>       described in [SASL] and/or the [DIGEST-MD5] mechanism.

I don't think we should be recommending DIGEST-MD5 anymore. Maybe SCRAM-SHA-256 instead?

> 6.3.9.  LIST Command

>       Note: The interpretation of the reference argument is
>       implementation-defined.  It depends upon whether the server
>       implementation has a concept of the "current working directory"
>       and leading "break out characters", which override the current
>       working directory.

Is there really any point in having the reference argument anymore? We could just require clients to use "" for it and simplify the protocol a lot?

> 6.3.10.  LSUB Command
> 
>    A special situation occurs when using LSUB with the % wildcard.
>    Consider what happens if "foo/bar" (with a hierarchy delimiter of
>    "/") is subscribed but "foo" is not.  A "%" wildcard to LSUB must
>    return foo, not foo/bar, in the LSUB response, and it MUST be flagged
>    with the \Noselect attribute.
> 
>    6.   Possibly fold in the following extensions/RFC: Base LIST-
>         EXTENDED syntax plus deprecate LSUB (replace it with LIST
>         \Subscribed) minus the requirement to support multiple list
>         patterns, STATUS-in-LIST, SEARCHRES, BINARY (only the FETCH
>         changes on leaf body part and make APPEND related ones optional.
>         See the mailing list discussion) - done.

I think at the very least we could remove the \Noselect kludge and return \Nonexistent instead.

> 6.3.11.  NAMESPACE Command
> 

>  If a client is configured such that it is required to
>    create a certain mailbox, there can be circumstances where it is
>    unclear which Personal Namespaces it should create the mailbox in.
>    In these situations a client SHOULD let the user select which
>    namespaces to create the mailbox in.

Is any client ever going to actually implement this? Maybe just change it to say to use the first personal namespace?

> 6.4.1.  CHECK Command

Time to drop this command?

> 6.4.5.  SEARCH Command

> In all search keys that use strings, a message matches the key if the
>    string is a substring of the associated text.
>    11.  Allow word-based searching (as per Chris Newman)?

Yes, I think we can't require substring matching anymore, since there's no practical way of implementing it efficiently for large installations. And many servers no longer support it.

> 6.4.6.  FETCH Command

>    ALL  Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE)
> 
>    FAST  Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE)
> 
>    FULL  Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE
>       BODY)

Time to simplify and drop all macros?

>    BINARY[<section-binary>]<<partial>>
>    BODY[<section>]<<partial>>
> 
>          The \Seen flag is implicitly set; if this causes the flags to
>          change, they SHOULD be included as part of the FETCH responses.

Time to simplify and require .PEEK versions? The auto-\Seen flag has just caused unnecessary complications.

> 6.4.10.  UID Command
>    Responses:  untagged responses: FETCH, ESEARCH

EXPUNGEs can also be returned by UID EXPUNGE.

> 7.4.2.  FETCH Response

>    RFC822  Functionally equivalent to BODY[], differing in the syntax of
>       the resulting untagged FETCH data (RFC822 is returned).
> 
>    RFC822.HEADER  Functionally equivalent to BODY.PEEK[HEADER],
>       differing in the syntax of the resulting untagged FETCH data
>       (RFC822.HEADER is returned).
> 
>    RFC822.TEXT  Functionally equivalent to BODY[TEXT], differing in the
>       syntax of the resulting untagged FETCH data (RFC822.TEXT is
>       returned).

Time to simplify and drop support for these?

>    CAPABILITY  Followed by a list of capabilities.  This can appear in
>       the initial OK or PREAUTH response to transmit an initial
>       capabilities list.  This makes it unnecessary for a client to send
>       a separate CAPABILITY command if it recognizes this response.

Could also mention that it is often sent in LOGIN or AUTHENTICATE tagged reply.

> 6.3.11.  NAMESPACE Command
> 7.2.5.  NAMESPACE Response

These are talking about "Namespace_Response_Extensions" but it's not explained anywhere. I've no idea what that is.

There are also some old imap-list discussions I had gathered in https://www.imapwiki.org/Specs/Rfc3501Problems - I think the relevant ones are:

 * LIST reply should never mix namespace separators. Different namespaces can have different separators, but in that case LIST shouldn't be listing those namespaces without explicitly being asked to.
 * Except INBOX is special and can have a different separator from others (I wouldn't mind dropping this though)
 * Should FLAGS / PERMANENTFLAGS be sent after a new keyword is created within the session
 * SEARCH TEXT includes searching MIME headers, while SEARCH BODY doesn't

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 25 Mar 2019 13:14:33 -0400

> Would result in mailbox-with-uidvalidity2 shrinking UIDVALIDITY from 2 to 1.

So the risk here is that you need to be very careful that you don't THEN rename back to '2' again ever, or a client that's been away for a long enough time can come back and find a reused UID. This is why "must always be greater" is a safer than "must not be any value which has ever been seen by a client in the past". Note I say "seen by a client" rather than "used" since the guarantees are only per client :)

> Also already in RFC 3501, but this MUST NOT has to be violated sometimes. For example otherwise it would allow client to FETCH a large message and keep reading it very very slowly over days, and se…

Yeah, plus middleware proxies. This is clearly bogus.

> * FETCH 1 (BODY[1] "NIL")

Agree that clarifying NIL behaviour would be great. The magicness of NIL and lack of quoting requirement on atoms is one of the horror cases of IMAP parsing.

> Is there really any point in having the reference argument anymore? We could just require clients to use "" for it and simplify the protocol a lot?

This I would also agree with :) The Cyrus server just concatenates the two, pretty much.

> I think at the very least we could remove the \Noselect kludge and return \Nonexistent instead.

Likewise this would simplify the Cyrus server... though again the problem is that if you want to remain backwards compatible is just increases the complexity of the server cases.

> Is any client ever going to actually implement this? Maybe just change it to say to use the first personal namespace?

There's a bug in Thunderbird where it creates in whatever namespace the mailbox name is a prefix of, which leads to it trying to create "Other Folders/Other" when the user types "Other". Ouch. Fricking namespaces.

On the other hand, SHOULD is still not a requirement, and most servers only have one personal namespace, so I'm pretty meh about this.

> > 11. Allow word-based searching (as per Chris Newman)?

There's already FUZZY.

> Time to simplify and drop all macros?

I would be fine with dropping macros.

> Time to simplify and require .PEEK versions? The auto-\Seen flag has just caused unnecessary complications.

Particularly since it's the default - I'd love to ditch the auto-\Seen, because right now in Cyrus it's done with an exclusive lock regardless of whether it finds anything to change... for sure that's acutally a silly idea and I _should_ have implemented it as an up-front scan followed by only exclusively locking if there was something to change.

(I'm pretty sure the auto-foo is done first so the modseq bump is returned in the fetch response)

> Time to simplify and drop support for these?

I'd have to update all my client code :p But yeah, besides it's an old name...

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 26 Mar 2019 09:45:57 +0100

On 25 Mar 2019, at 13:04, Timo Sirainen wrote:
> Also already in RFC 3501, but this MUST NOT has to be violated 
> sometimes. For example otherwise it would allow client to FETCH a 
> large message and keep reading it very very slowly over days, and 
> server wouldn't be allowed to disconnect it, because it can't send BYE 
> in the middle of the large mail literal.

Another example is if a client gets hung up (or goes silently offline) 
during a TLS negotiation. The server has to eventually timeout the 
connection but it would be semantically wrong to send a "* BYE" since 
there's no data channel. One case where this happened was when our SSL 
stack dropped support for an SSLv2-compatible hello (allowed by RFC 
6176), but some very old client kept sending such a hello (prohibited by 
RFC 6176). Incidentally, SSLv2 is completely incompatible with SSLv3 so 
the client sends something that's not sufficient to complete an SSLv3 
packet so both ends get stuck waiting until one side hangs up.

>> 5.1.  Mailbox Naming
>
>> servers MAY accept a de-normalized UTF-8 mailbox name and convert it 
>> to Net-Unicode prior to mailbox creation).
> and:
> This seems rather difficult from client's point of view. It CREATEs a 
> mailbox, but the client can't find it anymore because it changed. 
> Seems to me it would be less strange if the server simply failed the 
> CREATE entirely.

I had mistakenly thought this as well until I encountered the real-world 
situation. There are widely deployed mail clients that do not normalize 
Unicode when sending it to the server for mailbox names and also behave 
very badly from a UI perspective if CREATE fails. I tried an 
implementation which rejected CREATE when the client used a name in 
violation of 5198 and a customer insisted I provide a way to allow 
invalid names.

> If this is allowed for CREATE, is such conversion done for 
> SELECT/EXAMINE/APPEND as well?

I doubt this is necessary in practice. Most clients will take what they 
get from LIST responses and use that. Clients also have to deal well 
with SELECT/EXAMINE/APPEND failures because of reality.

> What about LIST "" <non-normalized name> - would it return the 
> normalized name?

I doubt most clients use that form of list, so I wouldn't implement that 
unless necessary.

		- Chris

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 26 Mar 2019 09:54:43 +0100

>>> 11. Allow word-based searching (as per Chris Newman)?

The issue is that when a server deployment becomes large and when it 
uses an indexed search technology (e.g., Solr, Elasticsearch, Lucene) 
then the server can not (and does not) perform body searches that comply 
with the IMAP specification. So I'd like the IMAP4rev2 search 
specification to be changed to reflect the reality that pure-substring 
search does not always match, especially for body searches. These 
technologies generally do word-based searches and can do word prefix 
searches reasonably efficiently, but can't do pure substring efficiently 
at scale. Also these technologies are often configured so that 
stop-words don't match (e.g., and, or, he, she, etc).

The function FUZZY performs is different and is an orthogonal issue.

		- Chris

---

From: Ned Freed <[email-masked]>
Date: Tue, 26 Mar 2019 07:13:06 -0700 (PDT)

> packet so both ends get stuck waiting until one side hangs up.

There was a suggestion (I think) during the meeting that this should be handled
by sending the TLS equivalent of "* BYE". Which sounds fine in theory, but as
said at the time, I'm skeptical that many/most TLS implementations provide
implementors with the means to conveniently do that.

				Ned

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 26 Mar 2019 16:42:38 +0100

On 26 Mar 2019, at 15:13, Ned Freed wrote:
>> >>    an untagged BYE response that contains the reason for having 
>> done
>> >> so.
>> >

I didn't hear such a suggestion. I made the observation that a wedged 
TLS negotiation is a scenario where sending "* BYE" shouldn't be done. 
It was suggested that example be mentioned in the document. I did not 
mention the fact that TLS has it's own alert syntax that could be used 
instead of "* BYE" -- I agree discussing that would be a bad idea.

		- Chris