Subject: [Lemonade] Discussion items from thsimorning on VFOLDER and WITHIN

From: "Stephane H. Maes" <[email-masked]>
Date: Thu, 1 Jun 2006 07:07:49 -0700

Here are the items that were discussed.
 
Stephane
 
PS: 
 
At todays FtF, there were some issues in VFOLDER discussed that we would appreciate some feedback in guidance from the WG on.

After the merge of VFOLDER and VIEW there are several parts of the draft which now offer optional behavior. Many members seem to think this is not desired and that Lemonade specs should offer clear and mandatory behavior to reduce client complexity. Assuming you agree with this, we need to decide which of the optional behaviors we want to be MUSTs and which to disallow. I will list them below:

1) RENAME - if a backing mailbox is renamed, what happens to views which are linked to this mailbox? Draft currently offers two options:

a) acts just like a DELETE of the backing mailbox, as far as what happens to vfolders dependent on that backing mailbox. Client/user must recreate vfolder and resync.

b) nothing happens. That is, the server handles updating all of the linked vfolders to use point to the renamed mailbox.

Randy put this as Windows (95) Shortcut semantics vs Mac Alias semantics. :) The FtF group seemed to lean towards the second option as it would reduce user confusion, lower customer support calls, and seems to be what people would want anyway. Anyone have any serious reasons why we should not mandate behavior b) ?

2) APPEND/COPY support for VFOLDER. Current merged draft says server can decide to reject an APPEND, or allow it.

There were arguments that perhaps it should be mandated to allow appends except in cases where ACL prevents it. Arnt proposed a case where a client might APPEND something, but the new message may not may any search criteria of the view, and thus some clients (particular those not VFOLDER aware) may make some bogus assumptions. Alexey pointed out that this could already be the case, since a secondary client connection could expunge the newly appended message before the appender gets to see it.

Opinions?

3) VFOLDERs based on dynamic message attributes. Old VFOLDER draft allows server to reject any VFOLDER creation if it deems search too expensive or invalid (with appropriate response code) Old VFOLDER draft disallowed dynamic attributes, VIEW allowed them. New VFOLDER+VIEW draft allows server to make the decision.

Randy pointed out that this would complicate some clients which might create vfolders without a user interface, such that a rejection of the search parameters by the search would mean many or all clients would have to have extra code to deal with this case (the client would not be able to prompt the user and ask them to rewrite the search based on returned error messages because the original search was not user authored)

So we have many options:

a) leave as it is.

b) mandate static message attributes support only, leave dynamic attributes for later draft

c) mandate dynamic attribute support

Opinions?

4) What are the requirements for when / how often a vfolder's search is evaluated? Some of the list of things which should trigger it: all agree on message deposit +

some options:

a) possibly any IMAP command (including NOP)

b) any IMAP command that modifies message state

c) on SELECT

d) periodically

I think everyone agreed that on message deposit, users will want to see this reflected immediately, but that it may be acceptable for some if flag changes or expunges are delayed.

5) The duplicate message/new UID problem. It is possible for some vfolder searches + sequence of operations to cause a message to appear in a vfolder, then disappear, and then reappear. As discussed currently, for this to work, when the message gets re-included in a vfolder it needs to be assigned a new UID for each time it is moved "to the front"

of the mailbox.

Randy suggested an alternative -- when a message goes "out of view", don't expunge it, leave it in the VFOLDER. Then, if a flag modification causes the message to be re-included, you don't need to generate a new UID, the client just resyncs the flags for that message.

What's the opinion on this? My only reaction is that some users might be surprised if they expect some messages to disappear out of view. For example, if a vfolder were to contain all messages which were flagged, and the user unflags the message, they might expect it to disappear from the vfolder.

_____
Stephane H. Maes, PhD,
Director of Architecture, Mobile, Voice, and Communications Platforms, Oracle.
Ph: +1-203-300-7786 (mobile/SMS); Fax / Office UM: +1-650-607-6296
e-mail: [email-masked]
IM: shmaes (AIM/Y!/skype) or [email-masked] (MSN Messenger) or [email-masked] (Google)

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 1 Jun 2006 17:54:13 +0200

Stephane H. Maes writes:
> 2) APPEND/COPY support for VFOLDER. Current merged draft says server 
> can decide to reject an APPEND, or allow it.
>
> There were arguments that perhaps it should be mandated to allow 
> appends except in cases where ACL prevents it. Arnt proposed a case 
> where a client might APPEND something, but the new message may not 
> may any search criteria of the view, and thus some clients 
> (particular those not VFOLDER aware) may make some bogus assumptions. 
> Alexey pointed out that this could already be the case, since a 
> secondary client connection could expunge the newly appended message 
> before the appender gets to see it.
>
> Opinions?

As I mentioned, this isn't a parallel. The case I fear is where the 
client has a loop like this:

decide which messages should be in the mailbox
until all those messages are in the mailbox:
    for each message not in the mailbox:
       append it

(Programs like imapsync contain code like this. Programs which copy mail 
en masse from one server to another.)

In Alexey's case, the message is appended twice. (Three times if two 
clients come along an expunge.)  In the case I'm afraid of, it's 
appended repeatedly until the backing mailbox explodes.

> 5) The duplicate message/new UID problem. It is possible for some 
> vfolder searches + sequence of operations to cause a message to 
> appear in a vfolder, then disappear, and then reappear. As discussed 
> currently, for this to work, when the message gets re-included in a 
> vfolder it needs to be assigned a new UID for each time it is moved 
> "to the front" of the mailbox.
>
> Randy suggested an alternative -- when a message goes "out of view", 
> don't expunge it, leave it in the VFOLDER. Then, if a flag 
> modification causes the message to be re-included, you don't need to 
> generate a new UID, the client just resyncs the flags for that 
> message.
>
> What's the opinion on this? My only reaction is that some users might 
> be surprised if they expect some messages to disappear out of view. 
> For example, if a vfolder were to contain all messages which were 
> flagged, and the user unflags the message, they might expect it to 
> disappear from the vfolder.

Randy's suggestion covers session-bound clients, but not offline 
clients. Consider a mobile client which connects three times, and the 
flag changes in the pauses between connections.

Arnt

---

From: Randall Gellens <[email-masked]>
Date: Thu, 1 Jun 2006 11:47:04 -0700

At 7:07 AM -0700 6/1/06, Stephane H. Maes wrote:

>   secondary client connection could expunge the newly appended 
> message before the appender gets to see it.

That seems very much an edge case.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The pure and simple truth is rarely pure and never simple.
                                            --Oscar Wilde

---

From: Randall Gellens <[email-masked]>
Date: Thu, 1 Jun 2006 12:02:47 -0700

At 5:54 PM +0200 6/1/06, Arnt Gulbrandsen wrote:

>> appear in a vfolder, then disappear, and then reappear. As 
>> discussed currently, for this to work, when the message gets 
>> re-included in a vfolder it needs to be assigned a new UID for 
>> each time it is moved "to the front" of the mailbox.
>>
>>  Randy suggested an alternative -- when a message goes "out of 
>> view", don't expunge it, leave it in the VFOLDER. Then, if a flag 
>> modification causes the message to be re-included, you don't need 
>> to generate a new UID, the client just resyncs the flags for that 
>> message.
>>
>>  What's the opinion on this? My only reaction is that some users 
>> might be surprised if they expect some messages to disappear out 
>> of view. For example, if a vfolder were to contain all messages 
>> which were flagged, and the user unflags the message, they might 
>> expect it to disappear from the vfolder.
>
> clients. Consider a mobile client which connects three times, and 
> the flag changes in the pauses between connections.

But if a message goes in and out of view, and gets a new UID each 
time it reappears, wouldn't it get downloaded multiple times?
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
We should never forget that everything Adolf Hitler did in Germany
was legal and everything the Hungarian freedom fighters did in
Hungary was illegal.               --Martin Luther King, Jr., 1963

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 1 Jun 2006 21:08:37 +0200

Randall Gellens writes:
> But if a message goes in and out of view, and gets a new UID each time 
> it reappears, wouldn't it get downloaded multiple times?

Yes. So it had better not happen often. Fortunately, I don't think it will.

If people use UNSEEN and SEEN in vfolders, and the \seen flag keeps 
being toggled on and off, then we have a problem. But the \seen flag 
typically is set once and that's it.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 01 Jun 2006 15:24:03 -0400

Hi Randall,

--On June 1, 2006 12:02:47 PM -0700 Randall Gellens <[email-masked]> 
wrote:

> But if a message goes in and out of view, and gets a new UID each time it
> reappears, wouldn't it get downloaded multiple times?

Yes. The current drafts says this:

   Future versions of VFOLDER may introduce a new server supported
   mechanism for efficiently determining the original mailbox and UID of
   a message

I think THIS version of VFOLDER should provide such a mechanism. A new OUID 
fetch parameter that returns a relative imap url with the mailbox, 
uidvalidty and uid of the underlying message would be one simple solution. 
The other would be a more general GUID mechanism - but that may be going 
too far...

Note that duplicate detection is only going to be done by clients that are 
aware they are dealing with a VFOLDER so there needs to be a select 
response to indicate that (this may have been discussed in the meeting - 
sorry if I missed it).

-- 
Cyrus Daboo

---

From: Dave Cridland <[email-masked]>
Date: Thu, 01 Jun 2006 23:28:16 +0100

Hi Cyrus,

On Thu Jun  1 20:24:03 2006, Cyrus Daboo wrote:
> Hi Randall,
> 
> --On June 1, 2006 12:02:47 PM -0700 Randall Gellens 
> <[email-masked]> wrote:
> 
>> time it
> I think THIS version of VFOLDER should provide such a mechanism. A 
> new OUID fetch parameter that returns a relative imap url with the 
> mailbox, uidvalidty and uid of the underlying message would be one 
> simple solution. The other would be a more general GUID mechanism - 
> but that may be going too far...

Now, I included this in my design for virtual mailboxes [everyone has 
done one of these, right?], but Arnt convinced me against adding it 
to his or V{FOLDER|MAILBOX}, because of the number of times this 
would be downloaded before it saved a message download.

Basically, it's a tradeoff, and if you have 100 messages, with a 100 
octet relative URL, that lands you with a ~10k bill, meaning that to 
pay for it, the duplicate body parts within those 100 messages must 
add up to more than 10k. Arnt convinced me that you're unlikely to 
read both backing and virtual mailboxes sufficiently for this to be a 
problem actually worthwhile solving.

Other options include using the message id, which ought to be unique 
at least for body parts if not headers, and is part of the Envelope, 
which good IMAP clients ought to be using anyway. This has the 
advantage that "mailing list duplicates" would be covered. (And 
although the mailing list copies actually differ, the body part 
differences don't matter and the extra headers are small).

For disconnected mode clients which aim to synchronize the entire 
mailbox, either method forces an extra round-trip, though, which 
depending on latency vs bandwidth may be less desirable.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 2 Jun 2006 12:10:27 +0200

Cyrus Daboo writes:
> I think THIS version of VFOLDER should provide such a mechanism. A new 
> OUID fetch parameter that returns a relative imap url with the 

I considered those, and concluded that it was likely to add more 
downstream traffic than it saves. URLs are big, and there are many 
GUIDs because they have to be sent in all mailboxes, not just vfolders, 
so there's a lot of bytes that have to be saved to come out ahead.

A third solution, which I thought of only when I read your message, is 
to add a new SELECT response with the backing mailbox and uidvalidity

     * OK [BACKINGMAILBOX mailbox nz-number] if you hafta ask...

and a new FETCH OUID to retrieve just the UID in that mailbox.

That saves traffic if the client saves just 1% of the 
(body[header.fields(...)] body[1]) fetches, perhaps even if it saves 
0.1%. For a client which issues FETCH OUID, that's not a bad bet.

Of course, this is troublesome wrt. vfolder-on-vfolder and also wrt. 
multi-mailbox vfolders.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 02 Jun 2006 09:23:29 -0400

Hi Arnt,

--On June 2, 2006 12:10:27 PM +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

> downstream traffic than it saves. URLs are big, and there are many GUIDs
> because they have to be sent in all mailboxes, not just vfolders, so
> there's a lot of bytes that have to be saved to come out ahead.

URLs have the advantage of only being needed in the vfolders and the client 
can use other sync methods to only fetch those for messages known to have 
changed or been added (i.e. it does not have to be part of the 'normal' 
fetch operations). You proposed shortcut below of sending the backing 
mailbox uidvalidity at select time would help cut down the length too.

> A third solution, which I thought of only when I read your message, is to
> add a new SELECT response with the backing mailbox and uidvalidity
>

OUID could return just the UID in the case of a single backing mailboxes, 
and in the case of multiple backing mailboxes, an alias for the mailbox 
name and the UID. The mailbox name could be given a short alias by the 
server and that could be included in the select response, then an even more 
compact form used:

* OK [BACKINGMAILBOX INBOX 12345 A] if you hafta ask...
* OK [BACKINGMAILBOX INBOX/sent-items 54321 B] if you hafta ask...
* OK [BACKINGMAILBOX INBOX/read-items 20606 C] if you hafta ask...
...
a FETCH 1:3 OUID
* FETCH OUID A:123
* FETCH OUID B:22
* FETCH OUID C:5432

etc

-- 
Cyrus Daboo

---

From: Dave Cridland <[email-masked]>
Date: Fri, 02 Jun 2006 17:12:09 +0100

On Fri Jun  2 14:23:29 2006, Cyrus Daboo wrote:
> OUID could return just the UID in the case of a single backing 
> mailboxes, and in the case of multiple backing mailboxes, an alias 
> for the mailbox name and the UID. The mailbox name could be given a 
> short alias by the server and that could be included in the select 
> response, then an even more compact form used:
> 
> 
I suspect you'd find this compressed to a similar length to the 
relative URL. After all, you're just doing a backreference, in 
effect. It might even compress worse, because of the extra framing 
used.

Dave.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 2 Jun 2006 09:37:32 -0700

At 9:08 PM +0200 6/1/06, Arnt Gulbrandsen wrote:

> being toggled on and off, then we have a problem. But the \seen 
> flag typically is set once and that's it.

I think this is client-specific, with some clients setting \seen once 
the message has been displayed to the user, or appears in a preview 
pane for a few seconds, or clicked past.  I think Eudora sets its 
internal \seen flag like this, and updates the server in batches.  I 
often reset messages to unseen, to let myself know that I haven't 
finished reading or thinking about a message.  So in my case, 
messages go between \seen and not often.

-------------- Randomly-selected tag: ---------------
There is a life-size picture of a dogcow conveniently located in the
Finder. Look under "Page Setup..."  Now look under "Options."  Like
any talented dog, it can do flips. Like any talented cow, it can do
precision bitmap alignment.                   --Apple Tech Note #31

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 5 Jun 2006 13:48:18 +0200

Something I just thought of.

An URL is sufficent to fetch a message, but not sufficient for a client 
to know whether it already has a message. This is because the server 
knows whether INBOX and users.daboo.inbox are the same, but the client 
does not. Ditto users.daboo.INBOX and users.daboo.inbox.

Of course, this is again just a few per cent lost. The client can do a 
case-sensitive equality test and be right almost all of the time, and 
the few per cent is just a minor slowdown in an unused case. Not worth 
much effort. (The common case being that the user reads either vfolders 
or the backing mailbox, not both.)

Arnt

---

From: Randall Gellens <[email-masked]>
Date: Fri, 9 Jun 2006 16:42:48 -0700

At 12:10 PM +0200 6/2/06, Arnt Gulbrandsen wrote:

>  A third solution, which I thought of only when I read your message, 
> is to add a new SELECT response with the backing mailbox and 
> uidvalidity
>
> (body[header.fields(...)] body[1]) fetches, perhaps even if it 
> saves 0.1%. For a client which issues FETCH OUID, that's not a bad 
> bet.
>
>  Of course, this is troublesome wrt. vfolder-on-vfolder and also 
> wrt. multi-mailbox vfolders.

Alexey pointed out (in a non-list discussion) that vfolder-on-vfolder 
can be done by storing the union of the selection criteria, meaning 
that there is really no such thing; that is, a vfolder-on-vfolder 
request is really just a client shortcut in creating a vfolder on top 
of a real mailbox.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The earth is like a tiny grain of sand, only much, much heavier.

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 11 Jun 2006 21:45:14 +0100

> but that may be going too far...

The server would also need to return OUIDVALIDITY (UIDVALIDITY of the 
original mailbox).

I think using relative IMAP URLs would be best.