Subject: RE: [lemonade] VFolder and IMAP extended LIST command

From: "Burger, Eric" <[email-masked]>
Date: Mon, 20 Mar 2006 20:11:53 -0500

By the end of the meeting, I was convinced of why we cannot just ignore
the problem.  My question for the list is: "Do we need a mechanism to
uniquely identify messages, so I don't have to fetch or store copies of
something I already have?"

In the meeting, we offered IMAP URL's.  Other alternatives are GOID
(Global Object ID [figment of my imagination]), a good hash, an opaque
token, etc.

Fundamental question: is this important?

-----Original Message-----
From: Dave Cridland [mailto:[email-masked]] 
Sent: Monday, March 20, 2006 6:24 PM
To: Arnt Gulbrandsen
Cc: support diverse service enivronments; Barry Leiba; Enhancements
Subject: Re: [lemonade] VFolder and IMAP extended LIST command

On Mon Mar 20 23:05:21 2006, Arnt Gulbrandsen wrote:
> Barry Leiba writes:
>> Cyrus brought up a question of whether it could be damaging (or 
>> resource-wasting, or user-confusing) to have vfolders appear as 
>> "regular" folders in a normal IMAP LIST.
> 
> In my experience, this doesn't happen. People don't do that.

The discussion was really relating to the specific issue that some 
desktop clients sync all message data in all visible folders, but it 
mutated to cover the more interesting case of whether mobile clients 
seeing the same message in multiple folders could be made to know it 
was the same message.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 21 Mar 2006 09:01:31 +0000

> Fundamental question: is this important?
Yes.

On a high-level overview, a mechanism like this means that clients 
can detirmine not only the same message appearing in multiple views, 
but the same message having been moved between mailboxes "out of 
sight" of the client.

But if you solve the problem by some form of identifier tied to the 
message then we get the interesting case where, given the GOID, we 
could add facilities to relocate the message - so we could use them 
as path components in IMAP URLs, perhaps.

Going further, ANNOTATE can be replaced by some kind of dedicated 
metadata store (like, say, WebDAV if you're trendy, or ACAP if you're 
not) which itself points to the message via these identifiers.

In principle, message-id fields could be used, but I don't know what 
the likelyhood of duplicate message-ids is in the wild. If we can use 
them to solve the problem in the majority of cases, that would have 
my vote. I was going to look into this (and content-id clashing, too) 
at some point, so I'll try to do this before Wednesday if there's 
interest.

To summarise, solving this problem correctly can quite easily yield 
gains in other areas.

Dave.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Mar 2006 10:52:16 +0100

Eric Burger writes:
> By the end of the meeting, I was convinced of why we cannot just 
> ignore the problem. My question for the list is: "Do we need a 
> mechanism to uniquely identify messages, so I don't have to fetch or 
> store copies of something I already have?"
>
> (Global Object ID [figment of my imagination]), a good hash, an 
> opaque token, etc.
>

I think not. But if it can be done easily and without mandatory 
bandwidth cost, it may be worthwhile. I'd favour a hash as a FETCH 
item. "UID FETCH 1234567 SHA256".

Arnt

---

From: Philip Guenther <[email-masked]>
Date: Tue, 21 Mar 2006 01:31:55 -0800

Dave Cridland <[email-masked]> writes:
...
>not) which itself points to the message via these identifiers.

A protocol exists where containers hold mappings from name to unique
ID, which may appear in multiple containers, and all actual data
access is by unique ID.  It's called NFS.

(Or at least NFSv2 and v3 are like that.  I haven't followed NFSv4.)

>the likelyhood of duplicate message-ids is in the wild.

The major problem with message-ids can be seen with this very
message.  You will receive two copies of this message, one direct
and one via the list.  They have different headers, at a minimum,
and I believe the latter will have its body altered as well.

Philip Guenther

---

From: Philip Guenther <[email-masked]>
Date: Tue, 21 Mar 2006 02:50:35 -0800

>access is by unique ID.  It's called NFS.

Sorry, I was too opaque by stopping there.  I think there are number
of parallels and/or warnings we can take from the file-sharing world
when considering server-unique IDs for message.  To blather some...

NFSv[23] use opaque binary blobs as file-handles.  Simply put, they
are they are returned by directory lookup operations and then passed
to most other operations include 'read' and 'write'.  (That's an
oversimplification; read the RFC or /usr/include/rpcsvc/nfs_prot.x
on a UNIX box for the details.)

In NFS, access rights are enforced on each object.  This mean that
just having a file-handle isn't enough: your authenticated ID must
be authorized to access the object.  This is what lets the access
control work in the face of multiple access-paths for a given
file-handle.  This contrasts to IMAP, where the only standardized
access-control is at the level of mailboxes, not messages.  Whether
you can access a given message _is_ dependent on the access-path.
So, for a server to decide whether a request for message with
GOID=142857 should be permitted, it has to track the reverse mapping
of which mailboxes that message appears in and have an efficient
way of tracking the union of rights.  That's a _large_ constraint
on server implementations.

Making the IDs persistent has its own problems.  As a protocol, NFS
does not guarantee persistence of file-handles past unmount or
across server restarts.  That contrasts with AppleTalk Filing
Protocol (AFP), which has always(?) supported persistent IDs and
used them for various things.  The result of that decision has been
to make basically impossible implementing a portable and _correct_
AFP server that shares files from an existing filesystem.  Some
fairly good implementations exist, but I believe the all fudge the
persistence aspect in some way.  Again, supporting the particular
requirement (persistent IDs) places a large constraint on the
backend.

Anyway, I've now swung from too succinct to blathering.  My point
is that this is heading into parts of the design space that are
both (partially) mapped and treacherous.

Philip Guenther

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 21 Mar 2006 11:54:00 +0100

> item. "UID FETCH 1234567 SHA256".

I should elaborate.

While implementing VIEW last autumn, we did some math, and couldn't see 
justification for this kind of download avoidance. There are many 
situations where it could help. Mailbox renames is a big one. But these 
situations just don't happen that often.

If an IMAP URL is used, then the URL may easily be 100 characters. Let's 
say it is, it's such a nice round number, and that the average message 
(body[header.fields(from to cc subject date)] body[1]) is 5000 bytes. 
Then fetching the URL in addition brings the total to 5100 bytes. In 
other words, a 2% extra cost, so the URL has to save more than 2% of 
message downloads in order to carry its own load. In my opinion, it 
should save substantially more, or else noone will implement it.

A hash is shorter. MD5 takes 20 bytes or so including syntax overhead, 
so it would pay for its traffic if it eliminates 0.4% of message 
downloads. Still not a clear win.

Arnt