Subject: Re: [Tzdist] tzdist examples

From: Daniel Migault <[email-masked]>
Date: Mon, 5 Jan 2015 15:43:33 +0100

Dear Friends and Colleagues,

Let me wish you a happy new year 2015. We are close to finalize the tz-dist
draft, and I expect it to be closed by next week!

To my knowledge, the only remaining issue is Issue # 32 on
history/changedsince. Cyrus needs some more feed backs from the WG to
determine whether changedsince should stay or should go. Please provide
your feed back by Thursday so Cyrus can provide the last version of the
draft. Most likely, this draft will go to WGLC as all issues have been
addressed.

Best Regards,
Daniel and Eliot

Here is the text from Cyrus that describes the use of changedsince for
synchronization.

So let me clarify by trying to compare the ETag vs changedsince behavior
for monolithic vs incremental version publishers:

1) Full sync /zones
   a) monolithic publisher: returns 500 items
   b) incremental publisher: returns 500 items

2) Poll to sync /zones + If-None-Match - nothing changed
   a) monolithic publisher: returns HTTP 304 status
   b) incremental publisher: returns HTTP 304 status

3) Poll to sync /zones?changedsince - nothing changed
   a) monolithic publisher: returns empty list response
   b) incremental publisher: returns empty list response

4) Poll to sync /zones + If-None-Match - assume one time zone changed
   a) monolithic publisher: returns 500 items
   b) incremental publisher: returns 500 items

5) Poll to sync /zones?changedsince - assume one time zone changed
   a) monolithic publisher: returns 500 items
   b) incremental publisher: returns 1 item

So 5(b) is a win for reducing the size of the response in the case where a
small number of time zones change when using an incremental publisher. Note
that all of the above options are supported by the protocol as it stands
today and can make use of intermediary caching for improved performance
during polling.

Whilst the IANA DB is currently using a monolithic version update scheme,
someone else could decide to republish that data using an incremental
scheme, e.g., by introducing their own per-time zone revision counters.

So I don't want to preclude the possibility of 5(b). So the question is can
we get consensus on whether or not the optimization that 5(b) represents is
worthwhile for the base tzdist specification. The answer to that determines
whether changessince stays or goes...

On Mon, Dec 29, 2014 at 2:48 PM, Ken Murchison <[email-masked]> wrote:

> On 12/26/2014 12:45 PM, Daniel Migault wrote:
>
>> My current understanding is that there is not a strong support for
>> keeping "changedsince" in the current spec, and that it could be left for
>> future work.
>>
>
> I wouldn't go that far.
>
> I agree with Cyrus that changedsince is important to reduce the size of
> the list response for clients looking to update the full set of time
> zones.  This is especially true now that we have added versioning
> information to the list response thus growing its size.
>
> Regardless of whether a client chooses to use changedsince, this
> functionality is fairly trivial to implement in a server, and 3 existing
> servers have already done so.  In fact, this functionality has been interop
> tested doing server to server time zone updates.
>
> --
> Kenneth Murchison
> Principal Systems Software Engineer
> Carnegie Mellon University
>
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Paul Eggert <[email-masked]>
Date: Mon, 05 Jan 2015 07:45:27 -0800

Cyrus Daboo wrote:

> So 5(b) is a win for reducing the size of the response in the case where a small
> number of time zones change when using an incremental publisher.

Yes, I can see that.  Currently there are 588 names and aliases in the TZ 
database, containing 8439 bytes total; if we assume 10000 bytes to represent all 
the names, then under the same assumptions as my earlier message 
<http://www.ietf.org/mail-archive/web/tzdist/current/msg01086.html>, a 
request-response pair without changedsince will be ((600 + 200) + (600 + 10000)) 
bytes for the initial request and 587 * ((600 + 200) + 600) no-change requests 
and 1 * ((600 + 200) + (600 + 4000)) change request, for a total of 838600 
bytes, which is pretty large and slow.  In contrast, 5b) shrinks this to ((600 + 
200) + (600 + 200)) + ((600 + 200) + (600 + 4000)) = 7000 bytes, which is 
considerably better.

The increased efficiency does not help small clients, as they're interested in 
only one zone anyway.  It helps only intermediate servers that are republishing 
the entire databases, or fat clients that always want the full database.  If we 
assume ten updates per year, the savings is (838600 - 7000) * 10 * 8 / (365.25 * 
86400) = 2.1 b/s per intermediate server.  So the question is whether a 2.1 
bits-per-second-per-fat-client optimization is worth complicating the protocol. 
  I can see cases where it might be.

This underscores, by the way, the need for a cheap way to download the entire 
database.  Since the entire tz database can be represented in 26 KB it still 
seems odd to me that we're spending all this time worrying about optimizing 
incremental updates.  But perhaps this can wait until v2 of the protocol.

> ETag does not allow for just the set of changed time zone meta-data to be
> returned in list action.

Sure it does: just ask for the changes since a previous ETag.  I think you 
mentioned this possibility earlier.  That is, the tzdist protocol needn't invent 
its own "dtstamp" attribute that is a confusing alternative to ETag; it can just 
use ETag.  That is, changedsince=XXX should use an ETag for XXX.

One other thing.  It should be acceptable for a server, for whatever reason, to 
ignore changedsince=XXX, and simply pretend that everything has changed and to 
list all the time zones.  This might occur when a server is replaced or its time 
zone publisher changes, or it could merely be a cheap server.  That is, the 
changesince= optimization should be optional on the server side.

With these two improvement to the protocol I think changedsince= would be OK.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 05 Jan 2015 07:51:07 -0800

Daniel Migault wrote:
> history/changedsince. Cyrus needs some more feed backs

I just sent some feedback.

There's one other remaining issue: there needs to be a standard way for tzdist 
to exchange tz-format binary data, as well as VTIMEZONE data.  Eliot Lear 
mentioned this in 
<http://www.ietf.org/mail-archive/web/tzdist/current/msg01171.html> as an issue 
he's considering opening.  This one really should be opened and then solved.  In 
practice right now the binary data format is undoubtedly more popular than 
VTIMEZONE in real-world clients, and standardizing its use in tzdist will make 
the protocol significantly more useful and more likely to succeed.

---

From: Lester Caine <[email-masked]>
Date: Mon, 05 Jan 2015 16:03:48 +0000

> draft. Most likely, this draft will go to WGLC as all issues have been

Am back in front of a proper desktop, but still working through 'paid'
traffic.

I still need to review a number of points, but I think this is still
being made a lot more complicated than it needs to be!

The whole problem is simply one of identifying that the information
being used is no longer valid. With the current OS based distribution,
applications do not get information that TZ has changed, so a service
that can flag 'changedsince' make sense. What also fails at the moment
is that the data provided via the OS updates is of variable quality. Be
that due to factors like binary data without leapseconds, or truncated
data where we have no idea just what the truncation is.

Create a reliable source that can be identified - publisher - add a
simple version mechanism - version - and we can at least have a little
more security that information normalized with a particular
publisher/version can be used reliably. Add the information that the
data is using version x but we now have x+1 which affects only a very
small subset of TZid's then one can establish if any of the data being
used MAY be affected. The BULK of the TZ traffic on tzdist need only be
diff sets of material.

Embedded devices and other services which don't need a full TZ dataset
will either use expand, or perhaps a single TZ dataset. If there is a
version change but it does not affect that TZid, the used 'version'
simply gets updated, while if the TZid IS affected the device has a
trigger to do something, but exactly what is up to the device. As a
minimum it would download the correct data ... and update version.

Over time as a publisher provides updates, each historic version
snapshot is simply a natural part of the process. As material is
archived a mechanism that records the publisher/version ensures that the
matching view of the TZ (and leapsecond) data can be recalled. It is not
putting any undue load on the server, simply providing a reliable way of
establishing the state of TZ when the data was created, and creating a
diff between the current version and the archived one will also come out
in the wash as it will only contain those TZid records that have
changed. Getting this process working from day one gets around the
current agro off having to reverse engineer the data from the new
complete data dump and make life very much easier for the multitude of
devices that currently have no means of even knowing a change has occurred.

Binary leapsecond correct data is simply a different wrapper around the
packet of TZid data. All of the embedded devices I'm currently looking
at do not use a seconds based calendar and it would be interesting to
know just how many calendar systems are day/time based rather than
'second'. There has been a debate in other lists over replacing the
second epoch based system with something a little more practical that
does not simply truncate the calendar at different points, and I know of
no archival system that uses seconds based calendars, using day/time
based binary data instead.

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 05 Jan 2015 11:16:07 -0500

Hi Paul,

--On January 5, 2015 at 7:51:07 AM -0800 Paul Eggert <[email-masked]> 
wrote:

> There's one other remaining issue: there needs to be a standard way for
> tzdist to exchange tz-format binary data, as well as VTIMEZONE data.
> Eliot Lear mentioned this in
> <http://www.ietf.org/mail-archive/web/tzdist/current/msg01171.html> as an
> issue he's considering opening.  This one really should be opened and
> then solved.  In practice right now the binary data format is undoubtedly
> more popular than VTIMEZONE in real-world clients, and standardizing its
> use in tzdist will make the protocol significantly more useful and more
> likely to succeed.

I think the simple answer to this is to define a MIME media type for the 
zoneinfo data format. That might be as simple as one new section in the 
tzdist document to do that (with a pointer to the IANA time zones page), or 
it may require a more detailed formal write-up of the data format (beyond 
the code) along with the media type. If the later, then I think it would be 
better as a separate spec.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Mon, 05 Jan 2015 17:34:40 +0000

On 29/12/14 13:48, Ken Murchison wrote:
>> keeping "changedsince" in the current spec, and that it could be left
>> for future work.
> 
> I wouldn't go that far.
> 
> servers have already done so.  In fact, this functionality has been
> interop tested doing server to server time zone updates.

While we are not concerned with 'how' a server does it's job, to get
around the existing problems with historic/archival data, moving away
from a 'single binary dump' of the TZ data to a versioned view of each
TZid allows many of the current holes to be plugged, and changedsince
does simply come out in the wash ... along with triggers to flag that
particular TZid's HAVE changed.

Part of the problem is that the TZ data is not reliant on any of the
current tz code, but it is that code which is being used to dictate what
should now happen, and that will not produce the 'right' result! An
interim solution that continues to support the current essentially
broken model will fit in fine, but providing a base for a proper
overhaul moving forward should be the aim here?

One point in relation to using 'seconds' based dates is of cause
identifying just what date range the calendar being used is truncated
to. Viewing 5000+ year old artefacts on Malta one needs a calendar that
will be accurate from -3500 BC right the way through with a clean
framework over the whole Roman pre and post BC period. TZ need only be
valid for the last three or four hundred years, but ideally one does not
then change the clock/calendar at some point, which is why a seconds
base is a potential problem, which using a different base avoids.

---

From: Doug Royer <[email-masked]>
Date: Mon, 05 Jan 2015 12:28:32 -0700

On 01/05/2015 07:43 AM, Daniel Migault wrote:

> determine whether changedsince should stay or should go. Please 
> provide your feed back by Thursday so Cyrus can provide the last 
> version of the draft. Most likely, this draft will go to WGLC as all 
> issues have been addressed.
>
My 2 cents worth:

I just need a way to tell if a TZ changes. I don't care if I have to get 
the entire TZ again or not. My code has no need for incremental change 
details or past out of date information.

  Anxiously awaiting the next riveting version and thanks for the hard work!

-- 

Doug Royer - (http://K7DMR.us / http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Tim Parenti <[email-masked]>
Date: Sat, 10 Jan 2015 20:05:29 -0500

On 5 January 2015 at 10:45, Paul Eggert <[email-masked]> wrote:

> mentioned this possibility earlier.  That is, the tzdist protocol needn't
> invent its own "dtstamp" attribute that is a confusing alternative to ETag;
> it can just use ETag.  That is, changedsince=XXX should use an ETag for XXX.
>

+1.  The only benefit I can see for using a timestamp here instead would be
if a client is switching between servers, but in that case the client
should be doing the "modified full sync" anyway, as outlined in #3 of
Cyrus' post at the head of this thread:
http://www.ietf.org/mail-archive/web/tzdist/current/msg01174.html

One other thing.  It should be acceptable for a server, for whatever
> reason, to ignore changedsince=XXX, and simply pretend that everything has
> changed and to list all the time zones.  This might occur when a server is
> replaced or its time zone publisher changes, or it could merely be a cheap
> server.  That is, the changesince= optimization should be optional on the
> server side.

+1; this is a very good point.

On 23 December 2014 at 13:24, Stephen Colebourne <[email-masked]>
 wrote:

> One final (optional) optimisation is that the ETag of the individual
> zone can be returned in the list response. This is effectively the
> "datatoken", but gives it a much more meaningful definition. Since
> making use of this requires querying the ETag from the clients HTTP
> cache, not all clients will be able to make use of it, but using the
> ETag of the zone response here is much more web-friendly than a second
> random "datatoken".
>

Also, +1 to further investigating Stephen's potential optimization.

--
Tim Parenti

---

From: Lester Caine <[email-masked]>
Date: Sun, 11 Jan 2015 11:46:51 +0000

On 11/01/15 01:05, Tim Parenti wrote:
>     Sure it does: just ask for the changes since a previous ETag.  I
>     think you mentioned this possibility earlier.  That is, the tzdist
>     protocol needn't invent its own "dtstamp" attribute that is a
>     confusing alternative to ETag; it can just use ETag.  That is,
>     changedsince=XXX should use an ETag for XXX.
>  
> +1.  The only benefit I can see for using a timestamp here instead would
> be if a client is switching between servers, but in that case the client
> Cyrus' post at the head of this
> thread: http://www.ietf.org/mail-archive/web/tzdist/current/msg01174.html

The bit I am still unhappy about is the continuing idea that the only
way tzdist will work is from a blank piece of paper? There is no need
for every user to have to download a complete set of data just because
one server has become inaccessible - perhaps temporarily.

There are a limited number of jobs to be done and how they are done is a
little academic, as long as one can identify some consistent element.

If version is part of the ETag element, then that mechanism can be used
as a part of the process. As long as the 'publisher' selection provides
a consistent set of current AND historic versions, then even a simple
'expand/TZid/period' can be server independent. If the primary server is
no longer available, reading just the expand request from an alternative
source will give the same response. changedsince in my book is simply
how the version element is managed and is needed to flag that we do want
to check for a change rather than a clean read. Embedded devices only
need to remember what publisher and version they are currently using for
perhaps a single TZid and never need a 'full sync' ... They will just
read a single TZid, or a rolling 'expand' and then 'changedsince' is
simply 'no' or a new set of data.

The examples in msg01174 are simply overkill in a lot of cases. If the
client already has 'publisher/version' data set up to a particular
version how ever it is sourced, there is no need to access tzdist other
than to check for a new version. That is either for a single TZid
dataset, or for the full data set, and an ETag or changedsince read of
the list of TZid's will return the same set of id's which ever server is
accessed. This is even more critical when in 50 years time someone is
looking at historic material which is correctly tagged with a publisher
and version. It's not something we can 'add in a later version'!

In the future a new 'publisher' may well become the standard, and that
may well require a complete new rework of all of the material including
mapping TZid's from some old model to a new one, but nothing we are
discussing now affects that process, and while historic archives MAY
need to be completely ported to some new format, access to the original
material also requires that the original publisher dataset is also
maintained.

---

From: Paul Eggert <[email-masked]>
Date: Sun, 11 Jan 2015 12:46:44 -0800

Lester Caine wrote:
> There is no need

Sure, and that's true regardless of whether list actions use changedsince=ETag 
or changedsince=timestamp.  If multiple redundant servers are well managed, they 
should agree about ETag values, just as they should agree about timestamps.

> as a part of the process.

That'd be one way to implement ETag values, but I'm leery of requiring this in 
the standard.  It'd be more fruitful to think of tzdist as something that at 
least in principle could be implemented atop standard caches like Squid and 
standard web servers like Apache.

---

From: Lester Caine <[email-masked]>
Date: Sun, 11 Jan 2015 22:08:20 +0000

On 11/01/15 20:46, Paul Eggert wrote:
> Sure, and that's true regardless of whether list actions use
> changedsince=ETag or changedsince=timestamp.  If multiple redundant
> servers are well managed, they should agree about ETag values, just as
> they should agree about timestamps.
> 
> That'd be one way to implement ETag values, but I'm leery of requiring
> this in the standard.  It'd be more fruitful to think of tzdist as
> something that at least in principle could be implemented atop standard
> caches like Squid and standard web servers like Apache.

I don't see any problem - as far as I am aware as long as the hash
returned via ETag for the same data package is the same for each server
then there is no problem, and certainly standard cache methods allow
that to happen? It is only that the accessing URL needs to provide the
current version in order to know that the server now has a new version
to provide, and more important if a server has not been updated an error
is provided if the replacement server does not actually understand the
version being requested.

The race problem may result in more than one version being propagated at
any time and this must not disrupt the process.

---

From: Paul Eggert <[email-masked]>
Date: Sun, 11 Jan 2015 16:34:17 -0800

Lester Caine wrote:
> It is only that the accessing URL needs to provide the
> to provide,

No, the ETag does not need to be part of the URL: it's a separate component of 
the protocol.  A client can use a conditional GET with an ETag to get either the 
latest version or an indication that the client already has that version.  That 
is how a client can know whether the server has a new version without putting an 
explicit version number in the accessing URL.

 > if a server has not been updated an error

Sorry, I'm not following this.  Are you thinking of a scenario where the client 
has a more-recent version than the server?  That's a problem, I suppose, but 
it's not a problem tzdist should try to solve.

---

From: Lester Caine <[email-masked]>
Date: Mon, 12 Jan 2015 10:09:31 +0000

On 12/01/15 00:34, Paul Eggert wrote:
> No, the ETag does not need to be part of the URL: it's a separate
> component of the protocol.  A client can use a conditional GET with an
> ETag to get either the latest version or an indication that the client
> already has that version.  That is how a client can know whether the
> server has a new version without putting an explicit version number in
> the accessing URL.

Something is getting lost in the translation from brain to paper ;)

If one looks up a page using a URL, the hash returned by the lookup as
the ETag will change if the underlying content is different. If I ask
for /expand/version/TZid ETag will be a fixed value which ever server
provides it. Use /expand/version/TZid/since then the ETag will return
the current version hash rather than requested version one.

Drop the 'version' element and one can't move between servers safely
since one does not know what is current at the alternate server. The URL
needs to  provide a version to work with, but also identify if it wants
the extra information that this version is not now the current one.

I THINK I am right when I say that if properly specified in tzdist, a
client can create a local ETag for all of the data it already has cached
and use that with the request for updates from the server without having
to have first read that data from a server? /list/version/since would
skip a download if the ETag of the local information matches the server
response?

> Sorry, I'm not following this.  Are you thinking of a scenario where the
> client has a more-recent version than the server?  That's a problem, I
> suppose, but it's not a problem tzdist should try to solve.

It is EXACTLY the race condition I am trying to avoid! If the diary I am
working with is built using a server that has yet to be updated. That is
the APPLICATION has yet to be notified that there has been a change, but
overnight I'm looking at a server that has already received the update,
my version is more recent than the one being used by my information source.

I can see the situation in the future where publishers trying to get
ahead of the game publish short term changes only to have the change
cancelled a few hours later. THAT is the sort of problem that
'changedsince' could iron out, since a TZid may appear in version+1 and
then go again in version+2 so a /expand/TZid/version/since would flag a
change of version number, but no change of content for that expand period.

/list/version/since where version is not available from a new server
being accessed would fail as invalid. This may be because the server has
not retained historic records, or because for some reason it has not
actually been updated. Both ARE potential problems which tzdist has to
handle.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 12 Jan 2015 08:44:32 -0500

Hi Lester,

--On January 11, 2015 at 10:08:20 PM +0000 Lester Caine 
<[email-masked]> wrote:

> then there is no problem

That may not happen. Remember that each server takes the raw publisher data 
and generates a representation in the media formats it servers up. For 
iCalendar, the generated data is not guaranteed to be byte-for-byte the 
same across all servers because different iCalendar libraries may generate 
data in slightly different ways, or generate different values for required 
properties like the product id.

We need to separate ETags (which represent a byte-for-byte state token for 
an HTTP resource) from the "semantic" state token - which in the case of 
tzdist is the tuple (publisher, version).

The (publisher, version) tuple is what clients use to sync across different 
servers - the ETag is what clients use to update from a server it already 
has data for. That is covered in the examples I sent out - in particular #3 
which shows a client with existing data from one server, trying to sync 
with a different server. I believe that covers exactly what you are after.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Mon, 12 Jan 2015 14:11:00 +0000

On 12/01/15 13:44, Cyrus Daboo wrote:
> That may not happen. Remember that each server takes the raw publisher
> data and generates a representation in the media formats it servers up.
> For iCalendar, the generated data is not guaranteed to be byte-for-byte
> the same across all servers because different iCalendar libraries may
> generate data in slightly different ways, or generate different values
> for required properties like the product id.
> 
> We need to separate ETags (which represent a byte-for-byte state token
> for an HTTP resource) from the "semantic" state token - which in the
> case of tzdist is the tuple (publisher, version).
> 
> The (publisher, version) tuple is what clients use to sync across
> different servers - the ETag is what clients use to update from a server
> it already has data for. That is covered in the examples I sent out - in
> particular #3 which shows a client with existing data from one server,
> trying to sync with a different server. I believe that covers exactly
> what you are after.

That all makes perfect sense, and I can understand the problems with the
format of information being different. However it is my understanding
that the generation of a value for ETag need not be strictly locked to
the physical data. Weak validation is specifically defined as
semantically equivalent and NOT byte for byte matches. If the data
packet has a semantic difference there something is fundamentally wrong
with that source?

I'm not bothered about just what ETag is doing, as long as there is now
agreement that (publisher, version) is the basis for something that is
flexible enough to meet some peoples alternate requirements while
allowing all of the currently identified race conditions to be avoided.
At the present time, as long as a publisher follows the basic rules I
can't see any possible holes in the future, even if a completely new TZ
identification structure came into existence, so if ETag says 'no
change' or a request returns an empty packet until such time as there is
new information then I'm not too bothered. Ideally both should simply
work? I think the argument is if ETag should be the only mechanism?