Subject: [VCARDDAV] Diff/UID/PID Issues

From: "Darryl Champagne" <[email-masked]>
Date: Thu, 29 May 2008 19:24:14 -0400

Hi,
In previous threads about UID/PIDs, there have been comments about storing
the original plus a diff, or using some form of revision control system.

Follows is an example to show where standard diff formats do not include the
necessary information, and why PID/UIDs are necessary.

Starting Point:

...10 lines of usual vCard stuff...
TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
TEL;TYPE=work:tel:+1-555-555-5555;ext=4444
TEL;TYPE=work:tel:+1-555-555-5555;ext=3333
TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
TEL;TYPE=home:tel:+33-01-23-45-67
...Usual vCard stuff...

As per the discussion of
http://www.ietf.org/mail-archive/web/vcarddav/current/msg00200.html, the
phone and/or server has additional information associated with each TEL
entry, such as the speed-dial number, or server information that is not
allowed to be shared with the phone (time of last call, sales info, etc.).
It is NOT information that is expected to be shared, or be useful with other
devices.

For this example, I will reference that additional info as:
SpeedDial numbers 11 - 15, which happens to match the line numbers.

Now I make the following changes:

1) Delete ext 4444, SpeedDial #12.

	$ diff -c2 t1.txt t2.txt
	*** t1.txt      Thu May 29 18:15:33 2008
	--- t2.txt      Thu May 29 18:21:14 2008
	***************
	*** 10,14 ****
	  Line 10
	  TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
	- TEL;TYPE=work:tel:+1-555-555-5555;ext=4444
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=3333
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	--- 10,13 ----

2) Change the preferred work number, extension 5555 with SpeedDial #11, to
extension 3333, and SpeedDial #13 to extension 5555.

	$ diff -c2 t2.txt t3.txt
	*** t2.txt      Thu May 29 18:21:14 2008
	--- t3.txt      Thu May 29 18:22:28 2008
	***************
	*** 9,14 ****
	  Line 9
	  Line 10
	! TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
	! TEL;TYPE=work:tel:+1-555-555-5555;ext=3333
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	  TEL;TYPE=home:tel:+33-01-23-45-67
	--- 9,14 ----
	  Line 9
	  Line 10
	! TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	! TEL;TYPE=work:tel:+1-555-555-5555;ext=5555
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	  TEL;TYPE=home:tel:+33-01-23-45-67

3) Change SpeedDial #11 to extension 2222, and SpeedDial #14 to 5555.

	$ diff -c2 t3.txt t4.txt
	*** t3.txt      Thu May 29 18:22:28 2008
	--- t4.txt      Thu May 29 18:25:40 2008
	***************
	*** 10,15 ****
	  Line 10
	  TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	- TEL;TYPE=work:tel:+1-555-555-5555;ext=5555
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	  TEL;TYPE=home:tel:+33-01-23-45-67
	  Line x
	--- 10,15 ----
	  Line 10
	  TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	+ TEL;TYPE=work:tel:+1-555-555-5555;ext=5555
	  TEL;TYPE=home:tel:+33-01-23-45-67
	  Line x

The question at this point, is if we can store just the original plus the
Diff, per earlier claims, how would that be done?

A simple diff provides:
	$ diff -c2 t1.txt t4.txt
	*** t1.txt      Thu May 29 18:15:33 2008
	--- t4.txt      Thu May 29 18:25:40 2008
	***************
	*** 9,16 ****
	  Line 9
	  Line 10
	! TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
	! TEL;TYPE=work:tel:+1-555-555-5555;ext=4444
	! TEL;TYPE=work:tel:+1-555-555-5555;ext=3333
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	  TEL;TYPE=home:tel:+33-01-23-45-67
	  Line x
	--- 9,15 ----
	  Line 9
	  Line 10
	! TEL;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	  TEL;TYPE=work:tel:+1-555-555-5555;ext=2222
	+ TEL;TYPE=work:tel:+1-555-555-5555;ext=5555
	  TEL;TYPE=home:tel:+33-01-23-45-67
	  Line x

However, this provides no information about which changes were edits, versus
deletes.  Unless the entire change history is stored, the necessary
information will not be available - which SpeedDial number goes with which
entry?

If I instead use a small PID value (or a large UID value) - in this case
equivalent to the original line number and SpeedDial number, I start with:

	...10 lines of usual vCard stuff...
	TEL;PID=11;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
	TEL;PID=12;TYPE=work:tel:+1-555-555-5555;ext=4444
	TEL;PID=13;TYPE=work:tel:+1-555-555-5555;ext=3333
	TEL;PID=14;TYPE=work:tel:+1-555-555-5555;ext=2222
	TEL;PID=15;TYPE=home:tel:+33-01-23-45-67
	...Usual vCard stuff...

Applying the same set of changes, the result/diff is:

	$ diff -c2 t1a.txt t4a.txt
	*** t1a.txt     Thu May 29 18:40:15 2008
	--- t4a.txt     Thu May 29 18:42:06 2008
	***************
	*** 9,16 ****
	  Line 9
	  Line 10
	! TEL;PID=11;TYPE=work,pref:tel:+1-555-555-5555;ext=5555
	! TEL;PID=12;TYPE=work:tel:+1-555-555-5555;ext=4444
	! TEL;PID=13;TYPE=work:tel:+1-555-555-5555;ext=3333
	! TEL;PID=14;TYPE=work:tel:+1-555-555-5555;ext=2222
	  TEL;PID=15;TYPE=home:tel:+33-01-23-45-67
	  Line x
	--- 9,15 ----
	  Line 9
	  Line 10
	! TEL;PID=11;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	! TEL;PID=13;TYPE=work:tel:+1-555-555-5555;ext=2222
	! TEL;PID=14;TYPE=work:tel:+1-555-555-5555;ext=5555
	  TEL;PID=15;TYPE=home:tel:+33-01-23-45-67
	  Line x

>From this, I can clearly see that SpeedDial 11 is now extension 3333,
SpeedDial 13 is now 2222, and SpeedDial 14 is now 5555.

Note that for the data shown in this example, I could have altered my
storage format such that it generated its own PID (including logic to create
new PIDs as necessary), and used that to store information in a version
control system, and figure out the actual changes to apply.  However, the
vCard Syntax without using a PID/UID is still not capable of showing what
changes to apply during synchronization (and how likely is it that the
people storing the data would know to do something like that).

Note that there is no requirement for the PIDs to be globally unique. If the
PID is defined as the Line number from the previous sync, or an incremented
value added to the Max Line number from the previous sync; that is
completely sufficient to carry the information needed.

e.g. sending a new record containing:
	TEL;PREVIOUS_LINENO=11;TYPE=work,pref:tel:+1-555-555-5555;ext=3333
	TEL;PREVIOUS_LINENO=13;TYPE=work:tel:+1-555-555-5555;ext=2222
	TEL;PREVIOUS_LINENO=14;TYPE=work:tel:+1-555-555-5555;ext=5555
	TEL;PREVIOUS_LINENO=15;TYPE=home:tel:+33-01-23-45-67
And for a new number, include:
	TEL;NEW_LINENO=101;TYPE=home:tel:+33-01-23-45-67

Which is more than sufficient for synchronization (or no PID at all for new
items).

Note that I am NOT advocating actually using the line number, or NEW and
PREVIOUS, as vCards are not ordered, and I expect that changing line numbers
would be very confusing.

If my server synchronized to 2 separate clients, each of which has added a
new number, then I am in the exact same situation as today - for every item
that did not exist in the previous sync with that client, compare both
sides, and take a best guess as to what it means.  I certainly know that my
NEW_LINENO=101 will be new to the client when I send it to the client, and
it does not necessarily have any correlation with the NEW_LINENO=101 coming
from this client (unless it is a slow sync, in which case I have to look for
duplicate records and IDs anyways).

Thanks,
dgc
--
Darryl G. Champagne
Technology Evangelist
Funambol
AIM/Y!/Skype: dgc03052
ph: +1 (603) 424-0323     Mob: +1 (603) 440-3621 (+ SMS)
[email-masked]          http://www.funambol.com 

_______________________________________________
VCARDDAV mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/vcarddav

---

From: Simon Perreault <[email-masked]>
Date: Fri, 30 May 2008 09:11:59 -0400

On Thursday 29 May 2008 19:24:14 Darryl Champagne wrote:
> storage format such that it generated its own PID (including logic to
> create new PIDs as necessary), and used that to store information in a
> version control system, and figure out the actual changes to apply. 
> However, the vCard Syntax without using a PID/UID is still not capable of
> showing what changes to apply during synchronization (and how likely is it
> that the people storing the data would know to do something like that).

I think this is a very good example, and it shows two things:

1. Diffs are only as good as the data that is diffed. If you want part of the 
data to not be visible in diffs, you need at least some kind of pointer, 
which the PID is. In this case I would consider the PID number as "a pointer 
to private data".

2. You don't want to share private data with other devices, but you do want 
other devices to keep track of your own pointer when making changes to shared 
data. If the PID somehow identified the "owner device", then there would be 
no conflict possible.

I have my little idea how this could be achieved simply, but let's agree on 
the basics first.
_______________________________________________
VCARDDAV mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/vcarddav

---

From: "Darryl Champagne" <[email-masked]>
Date: Fri, 30 May 2008 10:50:47 -0400

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Simon Perreault
> Sent: Friday, May 30, 2008 9:12 AM
> To: [email-masked]
> Subject: Re: [VCARDDAV] Diff/UID/PID Issues
> 

Just be careful not to attach to much significance to just the example.  It
doesn't matter if there actually is private data or not.  I might send a
vCard to someone where I have no private data.  Because they might have
private data at some point (including implicit data, like user interface
position), if I send an update I need to be able to identify which previous
specific item was changed, rather than just a new list of values.

The requirement includes that if 3 phone numbers are originally sent, and
two of them are changed, while one of them is deleted, the recipient of the
update be able to uniquely identify which one is which.

Start	   Change      End

Tel_1	   -> Tel_4    Tel_4
Tel_2    deleted
Tel_3    -> Tel 5    Tel_5

I need to be able to distinguish that Tel_2 was deleted, and Tel_3 changed,
rather than just having a new list of 2 numbers.

> data. If the PID somehow identified the "owner device", then there would

Because the private data might only be created in the future, the issue is
more the original sender of the data, not who actually has any private data.

And bandwidth is a big deal, and during a sync, a good DS server
implementation can identify what pieces it has that are different from what
the recipient currently has, and what is new from the recipient.  2 separate
vCards can be (usually are) exchanged - first the changes from one side,
then the merged set of integrated changes sent back.

---

From: Simon Perreault <[email-masked]>
Date: Fri, 30 May 2008 10:59:29 -0400

> specific item was changed, rather than just a new list of values.

Yes, but that can be cast into the private data model too. That implicit data 
really *is* data, exactly like a speed dial number.

> rather than just having a new list of 2 numbers.

I'm not sure what you mean. It looks like this could be accomplished with the 
diffs+PID combo of your example. Is there anything different to consider?

> more the original sender of the data, not who actually has any private

Well, until you do have private data associated with a property, you don't 
need to track which one is which, right? I think that's the central remaining 
question.
_______________________________________________
VCARDDAV mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/vcarddav

---

From: "Darryl Champagne" <[email-masked]>
Date: Fri, 30 May 2008 11:20:03 -0400

> Sent: Friday, May 30, 2008 10:59 AM
> To: [email-masked]

Yes, but below...

> diffs+PID combo of your example. Is there anything different to consider?

Simple, non-unique PIDs fully address this - without sending any new delta
format (in case you were heading there).

> need to track which one is which, right? I think that's the central

No - the sender has no knowledge of whether the recipient created private
data between the syncs or not.  It is just sending an update - but the first
vCard had to have already had identifiers for the update to be able to
preserve any potential private data.

Sending Tel_4 with a PID and Tel_5 with a PID is useless unless there
already were PIDs on TEL_1 etc.

And for DS we do _not_ want to send something like:
s/TEL_1/TEL_4/g
Because that requires remembering the exact prior state for every client
(which sometimes can get confused or lost - usually tossed out due to
aging), rather than just storing a simple integer with each current value.