Subject: [Tzdist] The version problem

From: Cyrus Daboo <[email-masked]>
Date: Thu, 18 Dec 2014 09:54:25 -0500

Hi folks,
Mike, Ken and myself have put together a proposal for changes to the 
current tzdist draft that we believe address the key issues surrounding the 
versioning problem that has been discussing on the list. This needs to be 
the main topic of discussion on the call (I think the Chairs will agree) as 
it is the major outstanding issue.

Here are the changes we are proposing:

1) The list action response "dtstamp" member is removed. It is replaced by 
a "synctoken" member. The value of that member is an opaque token that 
clients use with the "changedsince=" query parameter in a list action 
request. This addresses the problem of confusion over what the "dtstamp" 
value represents and the updated spec would make it clear that the 
"synctoken" values are unique to each server (i.e., use of a synctoken 
value from one server in a list action changedsince on another will result 
in a failure or a full list response).

2) A new section on timezone versions will be added to Section 3. This will 
describe two models of how time zone data is typically published. The 
"monolithic" model (which is basically the IANA/Olson model) is where the 
entire set of data is published under a single version identifier. The 
"incremental" model is one where each time zone has its own version 
identifier (or sequence or revision number) so that individual changes can 
be quickly identified.

3) The list action response will be changed to include "publisher" and 
"version" members for each object in the "timezones" array. This addresses 
the issue of provide explicit version information for the current set of 
time zones returned by the list response (to allow clients/application to 
handle issues such as race conditions). Note that the version number is 
either the overall "monolithic" version identifier, or the per-time zone 
identifier for the "incremental" mode (as described in the previous point).

4) The list action response is further changed to include a new "datatoken" 
member in each object in the "timezones" array (this can probably replace 
the "last-modified" member). The "datatoken" is an opaque token that is 
guaranteed to change only when the underlying time zone data changes. That 
change is independent of any changes to the "meta-data" (other items in the 
list response) associated with the time zone. i.e., when the IANA/Olson 
version changes from 2014a to 2014b, the "datatoken" for 
tzid=America/New_York remains the same, the "datatoken" for a time zone 
that did see a rule change between those versions , will be different. 
Adding this allows clients to skip download of unchanged time zone data for 
the "monolithic" publisher mode.

5) The specification will make it clear that the "changedsince" behavior is 
required to return any time zone whose meta-data or datatoken has changed. 
i.e., when the IANA/Olson version changes, the next changedsince response 
will include every time zone since the "version" member of every time zone 
is different. However, only a subset of time zones will have a different 
"datatoken". Whilst this does add to the size of the list/changedsince 
response compared to the current spec, the "datatoken" piece at least 
allows clients to be efficient with regards to download only changed time 
zone data.

We can discuss this in more detail on the call...

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Thu, 18 Dec 2014 15:11:51 +0000

> We can discuss this in more detail on the call...

I'm running out of time before I get on the road, and I'll not know the
internet access situation in the Hotel in Malta until I get over there.

As long as the 'synctoken' value CAN be used as a means of synchronising
with an alternate server, that is sync a compatible publisher set of
data then I think this does cover all the race conditions. For any
system I am building it will be the TZ version number but as I've
indicated already. I'm looking for the "incremental" model to transfer
changes. The "monolithic" model only applies if you have a number of
incompatible publishers using different TZid's to that managed via TZ in
my book, and the current version of TZ does not need to have been
supplied via tzdist.

Section 5 seems to be overly heavy. Since if I have asked for the
changes since the previous version only those changes are needed.
Currently that will be a list of TZid's but at some later version of the
protocol is could simply be a change to a generic TZ rule set which a
number of TZid's may be using. Once one has a proven version of the TZ
data, only the smallest of changesets is required to update it to the
next version. The part missing from the current TZ distribution process
is inly creating a 'changedsince' set.

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 18 Dec 2014 10:26:16 -0500

Hi Lester,

> my book, and the current version of TZ does not need to have been

"synctoken" is specific to one server only. What is "global" across all 
tzdist servers is the publisher/version information. If a device sync's 
with one server and then switches to another, it cannot re-use the 
"synctoken", so it has to do a full list action request. However, the 
response for each time zone will include the publisher/version details, so 
that client can compare with what it already has and decide whether to 
update or not.

Note that the version "model" (monolithic vs incremental) is determined by 
what the publisher is doing, not what the tzdist server does.

> changes since the previous version only those changes are needed.

Right - and if the publisher uses an "incremental" version model, where the 
version id is changed when the time zone data changes, then the list 
response will only return those time zones whose data changed. In the 
"monolithic" model, the list response will always end up returning all time 
zone meta-data, because the version id for every time zone has changed - 
clients can apply the meta-data changes to their entire cache. But clients 
can filter the list response using the "datatoken" to determine which time 
zones need their data refreshed. i.e., synchronization of the meta-data can 
be done independently of changes to the actual data.

> is inly creating a 'changedsince' set.

Yes there are a variety of optimizations that can be done as extensions if 
and when the need arises. Bit I think everyone is agreed that the first 
step is to just get implementors to start adopting what we have.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Thu, 18 Dec 2014 16:07:11 +0000

On 18/12/14 15:26, Cyrus Daboo wrote:
> response for each time zone will include the publisher/version details,
> so that client can compare with what it already has and decide whether
> to update or not.

OK this is my only sticking point then ...
The whole problem currently is that if one source stops being available,
one switches to another, but if all the sources are using the same base
TZ then there should be no need to have to download everything again.
You only need the incremental change ... as long as what is published
matches across all servers. Not actually a requirement of tzdist, but
I'm sure that in practice only the one clean view will eventually
evolve. Having different TZid's on different calendars and event simply
does not make sense.

> Note that the version "model" (monolithic vs incremental) is determined
> by what the publisher is doing, not what the tzdist server does.

For the monolithic model is there any reason NOT to be simply using the
raw TZ data? The whole point of tzdist should be to maintain a hopefully
consistent view of the TZ data and passing on updates in a timely manor.
A complete raw download should be available, but only as a backup
process, and knowing you have version X you should be able to ask any
server for X+1 ?.

---

From: Arthur David Olson <[email-masked]>
Date: Thu, 18 Dec 2014 12:04:51 -0500

> The "monolithic" model (which is basically the IANA/Olson model) is where
the entire set of data is published under a single version identifier.
> The "incremental" model is one where each time zone has its own version
identifier (or sequence or revision number)
> so that individual changes can be quickly identified.

The per-zone information might be an "earliest equivalent" monolithic
version number, or might be a list of equivalents.

    @dashdashado

On Thu, Dec 18, 2014 at 9:54 AM, Cyrus Daboo <[email-masked]> wrote:
> 2) A new section on timezone versions will be added to Section 3. This
> will describe two models of how time zone data is typically published. The
> 4) The list action response is further changed to include a new
> "datatoken" member in each object in the "timezones" array (this can
> probably replace the "last-modified" member). The "datatoken" is an opaque
> token that is guaranteed to change only when the underlying time zone data
> changes. That change is independent of any changes to the "meta-data"
> (other items in the list response) associated with the time zone. i.e.,
> when the IANA/Olson version changes from 2014a to 2014b, the "datatoken"
> for tzid=America/New_York remains the same, the "datatoken" for a time zone
> 5) The specification will make it clear that the "changedsince" behavior
> is required to return any time zone whose meta-data or datatoken has
> changed. i.e., when the IANA/Olson version changes, the next changedsince
> response will include every time zone since the "version" member of every
> time zone is different. However, only a subset of time zones will have a
> different "datatoken". Whilst this does add to the size of the
> list/changedsince response compared to the current spec, the "datatoken"
> piece at least allows clients to be efficient with regards to download only
> changed time zone data.
>
> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist
>

---

From: Daniel Migault <[email-masked]>
Date: Thu, 18 Dec 2014 19:37:35 +0100

Hi,

I feel that these versions address the uses case of Lester.

Regarding the different models I have a few questions:

Looks to me that "monolithic" and "incremental" models differs in that the
version applies to the whole set of TZ versus to each individual TZ.

Thus, could we define the monolithic model  as the model where all TZ have
the same version number. -- I am not sure that is not what Arthur said. The
inconvenient I see is that in that case a change of version for a given TZ
does not necessarily means a change has occurred. This would make device to
upload a TZ they already have.

Another alternative is to have different versions: the publisher-version
and the tz-version. I am just wondering if splitting these versions may
help to clarify the specifications/protocol.

BR,
Daniel

On Thu, Dec 18, 2014 at 6:04 PM, Arthur David Olson <
[email-masked]> wrote:
>
>
> > The "monolithic" model (which is basically the IANA/Olson model) is
> where the entire set of data is published under a single version identifier.
>> 1) The list action response "dtstamp" member is removed. It is replaced
>> by a "synctoken" member. The value of that member is an opaque token that

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Ken Murchison <[email-masked]>
Date: Thu, 18 Dec 2014 13:51:59 -0500

Hi Daniel,

On 12/18/2014 01:37 PM, Daniel Migault wrote:
> Looks to me that "monolithic" and "incremental" models differs in that 
> the version applies to the whole set of TZ versus to each individual TZ.
>
> Thus, could we define the monolithic model  as the model where all TZ 
> have the same version number. -- I am not sure that is not what Arthur 
> said. The inconvenient I see is that in that case a change of version 
> for a given TZ does not necessarily means a change has occurred. This 
> would make device to upload a TZ they already have.

This is where the proposed "datatoken" meta-data element for each TZ 
comes in.  Even in the face of an overall publisher version change, 
which we are considering a just meta-data change, an actual change of a 
zone's data is signaled by a change of its "datatoken".  The client only 
needs to download the TZ data for those zones whose "datatoken" has 
changed.  So between tzdata2014i and tzdata2014j, only the version 
meta-data element changes for America/New_York. The publisher and 
datatoken meta-data elements remain the same.

As was mentioned on the call, I think some examples in the draft will 
help clarify the various uses of publisher, version, synctoken, and 
datatoken.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Lester Caine <[email-masked]>
Date: Thu, 18 Dec 2014 19:17:09 +0000

> would make device to upload a TZ they already have.

If you do not have any data, then asking for version X provides a
complete set, while if you have a set then all you need is the changeset
for the new version. There are not two models since one NEEDS a full set
of data. All that changes is if you can update that data with a change
set, or need a complete set. As long a 'publisher' provides a consistent
view across ALL server sources, then asking if there is an update since
the version you have is all you need to process.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 18 Dec 2014 20:33:15 +0100

Thank you, Lester.  One of the outputs of today's discussion is that
Cyrus is going to write this into a draft so that people can get a feel
for how it would look more formally stated.  In addition he's going to
do up some examples.  I hope that this will bring both clarity and
closure to the situation.

Eliot

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 18 Dec 2014 14:47:26 -0500

Hi Lester,

> the version you have is all you need to process.

Well that is not the case. The tzdist server is providing a "view" of the 
publishers data, which includes possibly doing its own truncation as well 
as its own "interpretation" of the the data format used to return the data. 
e.g., in iCalendar, one could choose to return just a set of dates for 
observances, expanding out all the rules, or one could choose to use the 
rules. So the actual data returned by one tzdist server may be different 
from the one returned by another even though the same publisher/version is 
used as the source for both. That is always going to be the case since 
tzdist is not simply a "raw" dump of the publisher data.

So what does that means for the client? Well if the client got 2014j data 
from one tzdist server (or was provisioned with that through some other 
means), then we it gets to use a different server it has to do a full list 
action and look at the publisher/version data in that response. It is then 
up to the client to decide what to do:

1) It can choose to re-sync all the data from the new tzdist server so that 
it is fully in-sync with the data representation that server provides, even 
if the version matches what it has locally.

2) It can choose to sync just those items whose version is different from 
what it has locally. Yes, that means it will resync all the data when there 
is a "monolithic" version change - there is no way to get around that 
(other than to use a publisher with an "incremental" version model).

Now #2 is far from ideal - but in the absence of "incremental" version 
coming from the publisher that is the best we can do (without forcing the 
tzdist server to itself figure out all the incremental differences from all 
past versions).

-- 
Cyrus Daboo

---

From: Doug Royer <[email-masked]>
Date: Thu, 18 Dec 2014 13:36:31 -0700

On 12/18/2014 07:54 AM, Cyrus Daboo wrote:
> current tzdist draft that we believe address the key issues 
> surrounding the versioning problem that has been discussing on the 
> list. This needs to be the main topic of discussion on the call (I 
> think the Chairs will agree) as it is the major outstanding issue.
>
> Here are the changes we are proposing: ...

Looks fantastic, thanks for all the hard work.

Looking forward to the next draft push.

-- 

Doug Royer - (http://K7DMR.us / http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Lester Caine <[email-masked]>
Date: Thu, 18 Dec 2014 22:57:06 +0000

If the underlying TZ data is properly synchonised, which is the object of the exercise for the main servers, then I see no reason why once you have even a truncated list request the returned data would be different if the same request is made on any server. expand is a different case as I would anticipate different requests each time anyway, but if list/changedsince flags that the core TZid has changed one knows that expand may contain differences. 
I don't see any problem creating what ever range of truncated 'changedsince' set is requested, even an 'empty' responce if all of the changes are out of range. But I envisage tzdist updating the OS copy and so not truncated. The major problem currently is that the OS gets updated, but there is no trigger that there has been a change and as you say ALL the data gets replaced. What tzdist provides is a concise set of changes which can be used to check if your published data does need updating! 
If each application starts looking after a different set of data with different ranges then I think things will be cahotic.

Sent from my android device so the quoting is crap!

-----Original Message-----
From: Cyrus Daboo <[email-masked]>
To: Lester Caine <[email-masked]>, [email-masked]
Sent: Thu, 18 Dec 2014 7:47 PM
Subject: Re: [Tzdist] The version problem

Hi Lester,

-- 
Cyrus Daboo