Subject: [Tzdist] Using tzdist data for archival purposes

From: Lester Caine <[email-masked]>
Date: Wed, 27 Aug 2014 23:27:50 +0100

While trying to understand how things are expected to work I have been
only thinking about my own uses of the output of tzdist. Ideally I need
to be able to identify the source of the data being transfered, but the
reason for that has not been expanded. Cyrus's expansion on how he views
the system of caching will work prompted me to rethink what I was trying
to achieve, and the answer is quite simple. The whole object of my own
exercise is to establish a stable set of tz information along with being
able to track it's changes. This may in fact not even be possible via
tzdist. Please bear with me as this MAY already be handled in iCalendar
in some way I've not spotted yet?

The material I am working with is all archival, and has already been
normalized to UTC timestamps with a timezone flagging the local display
information. That timezone has a version, so that when reviewing the
data, the correct information is used. What I am trying to establish is
just how tzdist can be used to provide this data to other users. I had
initially thought I could use the dtstamp value as a means of
identifying the version of data used, and also to then pick up where the
data would require processing because of a change to that tz data. But
this fails simply because another user may not get the same dtstamp
values to work with. The only field that works here is the
'last-modified' tag as long as that is properly managed against the
source data and updates. *I* need a tzdist server which will return the
data for that particular 'last-modified' version to correctly recreate
the local data, and then a current version of the to calculate a new UTC
normalized entry. Since data will continue to build up for many years to
come, I don't currently see the need to check all of it every time there
is an update. That is only needed when data is accessed later, so the
version of timezone data used to store the data may be several years
behind what is currently being served up. Pulling a particular historic
view of the data is not a problem if it has been correctly archived, but
I am not sure that the ETag value can provide that reliably?

Moving this to a modern current example. I publish a schedule of
meetings for the next few years, across the world. These have a UTC base
so that video link schedules can be synchronised. They are using a
current version of timezone data. Some of the meetings WILL be affected
by Ramadan and other DST variations and we have already had practical
examples of the problem with video link meetings in recent years where a
site was an hour out of sync due to a missed DST change. People are
looking at copies of agendas on their own calendar systems using
different timezone sources, so relying on a site based synchronising
system would simply not work? We need to ensure that everybody is using
the correct data used to publish the schedule, and that also identifies
the problem created because of a change since that data was published.

Is that 'out of scope' of the current charter? If there is no provision
to ensure that people ARE seeing the same changes to data reliably
because we do not know if we have the same version of that data is this
actually solving the problem of timely distribution of changes?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Tobias Conradi <[email-masked]>
Date: Thu, 28 Aug 2014 06:18:39 +0200

The first two paragraphs I didn't understood what you were aiming at.
But reading the last two .... very interesting.

ETag uses time stamp and file content. Time stamp is irrelevant for sync.

Last-modified uses time stamp. Time stamp is irrelevant for sync.

What matters is the file content. Just use a normalized version of the
iCalendar data and calculate a hash, maybe excluding any last-modified
and restricting to a certain time period. If hashes match, then people
have the same data ready for usage. Whether they look at the same may
depend on localization. E.g. in IANA the acronyms are not unique and
users may see "IST" and don't know whether that is India or Israel.
But for the video conference example, where coordination of when it
happens is the focus, this hash would suffice.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Lester Caine <[email-masked]>
Date: Thu, 28 Aug 2014 08:45:28 +0100

> happens is the focus, this hash would suffice.

In ten ... twenty years time when accessing genealogical data we have
archived today, Will we be able to pull up the OLD timezone data that
goes with what has been actually been archived? We need to know that
what is contained in that archive actually matches the current
definition of the timezone ... more likely due to changes to the
historic data, but perhaps due to variations in more recent updates. For
my purposes I need to be able to load the matching view of tz that goes
WITH the data in order to THEN do a changedsince to the current view.
I'm trying to establish if ETag can be used in this context, but Cyrus
seems to imply that creating that should be left to the server software
rather than actually defining a reusable hash? In any case a date stamp
as well will indicate just how out of date material is. This bit is
probably out of scope? :(

( Just to put this in the context of my other ravings ... my involvement
in tz has been around both correcting historic material and ensuring
that the data available IS stable. Which is why it irritates that the
very data I'm trying to correct so MANX genealogical data is also
correct is simply being ignored as too much hassle :( )

Further in the concept of the modern problem. I publish the schedule
this morning ... there is an update due to DST changes in the middle
east ... the US update there tzdist server with that update when they
wake up, and look at the schedule ... They need to be in the Middle East
for X so they book flights ... only to find when I get notification of
the update and republish the schedule the meeting is an hour earlier!
Relying on 'file content' is the problem here if there is no mechanism
to ensure that the RIGHT file content is being used!

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 07:47:21 -0700

> to ensure that the RIGHT file content is being used!

It sounds like your needs will be met if we add a Section 5.x (between 
Sections 5.2 and 5.3) that is something like this:

'5.x.  "dbversion" parameter

Name: dbversion

Description: Specify the time zone database version for the request.

Value: A string representing the time zone database version.  It should 
be the string contained in the "dbversion" object returned in a server's 
previous "capabilities" response.  If this parameter is not specified a 
server default is used.'

and append the following text to Section 5.3:

'If the "dbversion" parameter is also given, the "changedsince" value is 
ineffective, since a specific database version SHOULD never change.'

and then modify Section 6 (which I haven't yet read) accordingly.  For 
example, if the server is using unmodified tz data and dbversion is 
"2014e" you'd get the tz database version released on June 12 of this 
year, if it's "2014e" you'd get the May 27 version, etc.  Publishers and 
providers can use their own version number, e.g., the machine I'm typing 
this on would use version "2014e-0ubuntu0.14.04".  If a server wants to 
support only its latest version, it could reject requests to use earlier 
versions.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 28 Aug 2014 10:55:38 -0400

Hi Paul,

> support only its latest version, it could reject requests to use earlier

First off I don't think this needs to be part of the base tzdist protocol - 
adding it as an extension, once all the use cases are fully understood, 
would be better. That said, I think if we try and do this we would end up 
inventing our own version control system (VCS) in tzdist (e.g., some users 
might want to see just the "diffs" between two version, or perhaps a 
timezone data base "forks" so there becomes a need to track "branches"). A 
better solution might simply be to have a capability item that points to an 
established VCS system (subversion, git, WebDAV versioning, etc) hosting 
the full time zone history.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 08:13:13 -0700

Cyrus Daboo wrote:
> A better solution might simply be to have a capability item that points
> to an established VCS system (subversion, git, WebDAV versioning, etc)
> hosting the full time zone history.

Wouldn't that entail an undesirable race condition between the time that 
a client fetches the capability item and the time that the client 
fetches actual data?

Of course tzdist should not invent a version control system, or to 
support diffs or forks, or whatever.  However, a database version string 
should be easy for a server to support independent of version control 
technology.  The database version string could be mapped via a 
capability item to the more-detailed info that you suggest.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 28 Aug 2014 11:41:20 -0400

Hi Paul,

--On August 28, 2014 at 8:13:13 AM -0700 Paul Eggert <[email-masked]> 
wrote:

> a client fetches the capability item and the time that the client fetches
> actual data?
>
> technology.  The database version string could be mapped via a capability
> item to the more-detailed info that you suggest.

Well, OK, but I think that was the purpose of the existing "primary-source" 
though perhaps that appears to be more of a "descriptive" representation of 
the version. Perhaps we need a bit more structure here - i.e., 
"primary-source" is the descriptive text, say just "IANA TZ DB", and 
"dbversion" is the version id "2014f". The question is who gets to decide 
what the version number is? Do we require all publishers to tag their 
release with a version in some obvious way (more than just encoding it in 
the file name)? Is it up to the provider to decide on their own version 
tokens?

-- 
Cyrus Daboo

---

From: Tim Parenti <[email-masked]>
Date: Thu, 28 Aug 2014 13:35:45 -0400

On 28 August 2014 10:55, Cyrus Daboo <[email-masked]> wrote:

> First off I don't think this needs to be part of the base tzdist protocol
> - adding it as an extension, once all the use cases are fully understood,
> would be better.

I've always seen this as a problem properly handled by the software using
the zone data.  If a change to zone data is detected that affects one or
more events, any given user might wish to:

   1. Keep the event(s) at the same wall-clock time, changing the
   corresponding UTC time.
   2. Keep the event(s) at the same UTC time, changing the corresponding
   wall-clock time.
   3. Keep the event(s) at the same time relative to some other zone, which
   may or may not change the corresponding UTC or wall-clock times.
   4. Defer this decision to someone else.

For most individuals not travelling across zones and not sharing events
with others, I'd expect the answer to typically be (1).  But as events are
shared across zones, (2), (3), and (4) become more attractive options,
depending on the circumstances.

I'm not convinced that it is within our scope to solve these problems; only
to distribute the data so that end-users can make informed and timely
decisions.  To that end, we could suggest that calendaring implementations
using data via tzdist not handle such changes silently and actually inform
the end-user.

--
Tim Parenti

---

From: Tim Parenti <[email-masked]>
Date: Thu, 28 Aug 2014 13:38:55 -0400

On 28 August 2014 11:41, Cyrus Daboo <[email-masked]> wrote:

> Perhaps we need a bit more structure here - i.e., "primary-source" is the
> descriptive text, say just "IANA TZ DB", and "dbversion" is the version id
> "2014f". The question is who gets to decide what the version number is?

If the primary-source is merely descriptive, so too is the dbversion, I
think.  It would be up to each tzdist provider to either use a
version-tagging scheme adopted by each upstream data provider, or to
develop its own.

--
Tim Parenti

---

From: Mike Douglass <[email-masked]>
Date: Thu, 28 Aug 2014 14:03:01 -0400

On 08/28/2014 01:35 PM, Tim Parenti wrote:
>
> On 28 August 2014 10:55, Cyrus Daboo <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     First off I don't think this needs to be part of the base tzdist
>     protocol - adding it as an extension, once all the use cases are
>     fully understood, would be better.
>
>
> I've always seen this as a problem properly handled by the software 
> using the zone data.  If a change to zone data is detected that 
> affects one or more events, any given user might wish to:
>
>  3. Keep the event(s) at the same time relative to some other zone,
>     which may or may not change the corresponding UTC or wall-clock times.
> For most individuals not travelling across zones and not sharing 
> events with others, I'd expect the answer to typically be (1).  But as 
> events are shared across zones, (2), (3), and (4) become more 
> attractive options, depending on the circumstances.
>
> I'm not convinced that it is within our scope to solve these problems; 
> only to distribute the data so that end-users can make informed and 
> timely decisions.  To that end, we could suggest that calendaring 
> implementations using data via tzdist not handle such changes silently 
> and actually inform the end-user.
>
Absolutely. We can't make any assumptions from the data we can see. Just 
because a user only has local time events doesn't mean they don't travel 
or that those events don't depend on UTC - for example I have events 
with no attendees, referring to telephone calls involving other 
countries. Also, after a DST change, events may conflict with other 
events not on that particular calendar.  The only safe option is to 
inform the user and to reschedule meetings.

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 11:06:05 -0700

Cyrus Daboo wrote:
> that was the purpose of the existing "primary-source" though perhaps
> that appears to be more of a "descriptive" representation of the version.

You have the advantage of me, since you've read Section 6 and I haven't. 
  :-)

That being said, if my brief perusal of Section 6 is right, the 
"primary-source" capability won't work, since it allows race conditions 
when the database version changes between one response and the next.

However, doesn't the ETag required by Section 6.3 partly satisfy Lester 
Caine's needs?  Suppose, for example, that a server providing vanilla 
2014f data outputs 'Etag: "2014f"' in the header of every HTTP response. 
  Then clients will know whether the database has changed between two 
responses.  And an extended protocol could allow a "dbversion" request 
parameter so that a client could specify the desired database version as 
an etag value.

I suppose another interpretation could be that each response could have 
a different ETag, even if the two responses come from the same databasse 
version; but that interpretation would be less useful for clients, and I 
don't see why we need to allow it.

---

From: Cyrus Daboo <[email-masked]>
Date: Thu, 28 Aug 2014 14:54:39 -0400

Hi Paul,

--On August 28, 2014 at 11:06:05 AM -0700 Paul Eggert <[email-masked]> 
wrote:

> I suppose another interpretation could be that each response could have a
> different ETag, even if the two responses come from the same databasse

We can't dictate what goes in the ETag value - that is fundamental to HTTP 
itself and different server implementations will do things differently.

Perhaps the proper option is to have "dbversion" returned in the 
action=list response. That way a client using list to find changes also 
gets told the current dbversion as part of that response.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 12:07:47 -0700

> gets told the current dbversion as part of that response.

That should work, though we'd need a "dbversion" in responses to other 
actions too.  Otherwise (given what you say about ETag) a client can't 
determine whether the retrieved time zone data are all consistently from 
the same database version, and so cannot reliably check the provenance.

---

From: Lester Caine <[email-masked]>
Date: Thu, 28 Aug 2014 23:21:50 +0100

> inform the user and to reschedule meetings.

The current system of distribution has a problem with identifying when
this particular problem arises. FIXING the resulting problem is not a
problem, it's the fact that the problem gets missed due to the poor
quality of the distribution system. I would hope that it is tzdist
intention to FIX the problem otherwise there is no improvement to the
poor way urgent DST updates are distributed!

I'll pick up other points in detail in other messages.

---

From: Lester Caine <[email-masked]>
Date: Thu, 28 Aug 2014 23:41:20 +0100

> the same database version, and so cannot reliably check the provenance.

The way ETag is being documented to work makes it totally useless as a
way of synchronising data across a NETWORK of servers. The MAIN problem
we have today is that the clients are using data from different sources
with different levels of both accuracy and update. The charter for
tzdist specifically states "A service is needed that can provide timely,
reliable updates.". While the ETag mechanism may well ensure a single
server is providing reliable updates to it's own clients, the
reliability of that process simply does not scale to 'vast numbers of
clients' where even just a second secondary server is added? There is
simply no guarantee that information used to create data on one server
can be identically read on a server on the other side of the world?

Now this may be 'out of scope' but I would contend that NOT providing a
service which achieves the simple stated aim of an efficient timely
update to ALL of the clients is of little use to anybody? That every
secondary server is a race hazard makes things worse not better?

---

From: Lester Caine <[email-masked]>
Date: Thu, 28 Aug 2014 23:53:32 +0100

> don't see why we need to allow it.

Etag is undefined across different servers even if they are serving
exactly the same version of the underlying data. Some other mechanism is
required for a user to know they are using the same version of data as
that which was used to create then material they are trying to use. The
only way ETag could be used is if every client using the data is using
the exactly the same server as the originator ... which in my book
breaks the 'protocol must scale to vast numbers of clients' requirement
of the charter. ETag is simply not scalable UNLESS it's generation is
locked down to return the same value for the same version of data.

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 16:56:45 -0700

Lester Caine wrote:

> exactly the same version of the underlying data.

Although that's true for vanilla HTTP, it need not be true for the 
tzdist protocol, as (by definition) the tzdist protocol consists of 
further restrictions on HTTP.  So, for example, the tzdist protocol can 
say, "Use HTTP, except with the restriction that every 'get' response 
MUST include an ETag response header" -- a restriction that is already 
present in the draft.

Similarly, the tzdist protocol could impose the further restriction that 
every ETag MUST start with the letter 'Q'.  That would be a silly 
restriction, but it's certainly doable.

And similarly, the tzdist protocol could impose the extra restriction 
that the ETag MUST be the dbversion string.  Nothing in the HTTP or 
draft tzdist protocol would prohibit our adding such a restriction.  And 
this would fix the problem.

---

From: Lester Caine <[email-masked]>
Date: Fri, 29 Aug 2014 01:55:45 +0100

> this would fix the problem.

My point here was that Cyrus has already stated that he does not want to
specify how the ETag is created. He wants to leave that to the
implementers without restriction. YES making it prescribed helps build a
more reliable system :)

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 22:01:26 -0700

Lester Caine wrote:
> implementers without restriction.

OK, but that's not incompatible with being able to deduce provenance 
from the ETag.  For example, tzdist could provide a request in which the 
client supplies an ETag (that was earlier returned by the server) and 
the server responds with the corresponding dbversion; that should suffice.

But to be honest, I don't understand why tzdist must leave choice of 
ETag spelling to the implementers without restriction.  We can certainly 
impose *some* ETag-related restrictions, so why not impose a restriction 
that would really help with provenance?

A related question: what's the difference between a tzdist local 
provider and an HTTP cache?  That is, why not have tzdist reuse RFC 7234 
instead of trying to reinvent it?  If we do need to reinvent RFC 7234, 
the draft should discuss why HTTP caches don't suffice.

---

From: Lester Caine <[email-masked]>
Date: Fri, 29 Aug 2014 08:35:33 +0100

> the server responds with the corresponding dbversion; that should suffice.
The problem I have with that is if the material being processed was
created with one server how does a user on a different 'branch' of the
distribution system get the same dbversion back?

> that would really help with provenance?
And that I think plugs my hole. It does mean the distribution system
needs to be able to supply historic material, and some will view that as
outside the scope, but even just the previous version of a data set may
be required to ensure a consistent view of what is being distributed.

In reality all *I* am looking for is that 'changedsince' simply flags up
that there is a change which MAY have a an effect on what the user is
looking at. The note in tz would indicate that - perhaps - the exact
timing of the next transition may change. Quality of data is out of
scope, but this particular uncertainty in future data IS causing
problems today! I'm not looking to 'automaticall fix it - just stick a
hand up.

> the draft should discuss why HTTP caches don't suffice.
That is something I will need to read ...

---

From: Paul Eggert <[email-masked]>
Date: Fri, 29 Aug 2014 05:07:48 -0700

> distribution system get the same dbversion back?

As long as the users have the same provider they should be fine because 
its secondaries will use the same ETags, and that should be enough. 
Although it'd be nice to have ETags be "universal" to all providers, 
it's not necessary.

It's like dbversion.  We don't need to insist on world-wide unique 
dbversion strings; all a client needs is a dbversion string that's 
unique to its provider.  That's why it would work to use the dbversion 
as the ETag.