Subject: [Tzdist] Test Server

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 11:10:03 +0100

Please can I request that the Test Server gives the correct
'capabilities' report. The Data being served up is the 1970 truncated
version of tz, and so the capabilities should reflect that. While the
data does contain some pre-1970 material, the aliases contain
differences provided by the backwards file, so looking up an alias does
not give reliable data. The only valid truncate point is 1970 when
ignoring the backward data. This does then beg the question as to if the
pre-1970 data should be included at all? Once tagged as truncated, then
only post 1970 data should be included? Alternatively the backwards file
could be included, and all the aliases properly supported.

The sample get and expand entries seem to have an extra \ in front of
each & which I seem to recall is a wiki problem escaping characters.
Strip them manually on the address line and the demo's work.

I am looking at the data returned for the Isle of Man which is only
different in it's start date to that returned, but the way the rules are
converted from those supplied in tz do not make for easy processing when
trying to check for changes. I can see now why this does not make easy
processing when making changes to the generated iCalander data due to a
change in a tz rule. Does anybody have a documented set of rules for
doing the translations? I know the answer is 'it's out side the scope of
tzdist' and as far as many of you are concerned, only the resulting
generated offsets matter, but it is the handling of changedsince which
matters and if the process of checking throws up false updates due to
the way the data has been processed this may be a problem.

I think from my own point of view, passing the timezone data as a json
packet may be the correct way forward for cloning the data. That some
rules are merged and others are expanded in VTIMEZONE does not make for
easy update management. Just as with the missing data described above,
large sections of the data are shared, and there is no need to
unnecessarily duplicate large sections of common rules. Just allow the
timezone definition to 'include' a common rule set.

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Ken Murchison <[email-masked]>
Date: Mon, 25 Aug 2014 09:40:48 -0400

> could be included, and all the aliases properly supported.

Ok, I'm confused and probably ignorant.  In the third sentence you say 
that the returned data is incorrect because it includes aliases from the 
backwards file, but in the last sentence you say that the data could be 
made correct by including aliases from the backwards file.  Can you 
please give a specific example of an incorrect timezone resource and its 
aliases?  I'm sure my server is returning data quite similar to 
calendarserver.org

> the way the data has been processed this may be a problem.

changedsince as currently defined can and will change if the iCalendar 
data of the resource changes, regardless of whether it changed because 
the underlying tzdata changed or because the translator being used 
constructed the iCalendar data differently. dtstamp and changedsince are 
only used by the client to fetch the latest version of iCalendar 
resources or a range of expansions.  Its not designed for version 
history or anything along those lines.  Its really the same as the HTTP 
Last-Modified response header, but with the advantage that when used 
with the list action it allows a client to tell the server to show it 
ALL resources which have changed since the given timestamp in one 
roundtrip rather than forcing the client to check all of the resources 
individually (each with its own roundtrip).

> timezone definition to 'include' a common rule set.

If someone would like to use the tzdist protocol to pass around 
JSON-based representations of the tzdata rules, then an 
application/tzdata+json format can be designed.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 15:45:02 +0100

> aliases?  I'm sure my server is returning data quite similar to

Currently the additional timezones contained in backzone are absent from
the returned data The zones are returned as their post 1970 match. All
of the UK  satellite timezones have different pre 1970 data.
America/Port_of_Spain is a set of several merged zones
Europe/Belgrade combines different data.
Current backzone file has over 50 extra sets of details, but only a
small amount of actual changed data.

Option one ... Identify the data as truncated - since it has been  built
from the truncated view.
Option two ... add all the extra timezones and remove those as aliases
in 'list'

> individually (each with its own roundtrip).

Indicating to a user that something has changes IS all about version
control ... and needs to be able to cope with the whole period that the
server has been supplying data which may have been cached using dtstamp
any time during the period, and then return a correct response to a
changedsince request against that dtstamp. The un-answered question is
if changedsince will give a positive response if there has been changes
to the underlying data in the intermediate period which do not actually
result in a difference for the particular enquiry posted.

> application/tzdata+json format can be designed.

I am just saying here that VTIMEZONE is probably not the most efficient
method of mirroring the core data on secondary servers. All of the
backzone data could probably be added in a few extra lines while the
VTIMEZONE requires a full expansion of every one :(

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 25 Aug 2014 11:23:14 -0400

Hi Lester,

> small amount of actual changed data.

I am not sure where the "1970" validity is coming from here. Let me explain 
what I do to get the data for our test server:

1) I download the current tzdb from IANA (though the current test server is 
a few releases out of date)
2) I run my own tzdb->iCalendar conversion tool (open source available 
here:<https://svn.calendarserver.org/repository/calendarserver/PyCalendar/trunk/> 
on the "zonal" python package)
3) That tools scans the primary definition files for all "Zone" entries and 
generates a VTIMEZONE for each of those using all the specified rules.
4) For each "Link" entry in the main files and those in the "backward" file 
it generates a duplicate VTIMEZONE from the linked original, but using the 
link name for the tzid.

The tool does not use the information in either of the zone.tab or 
zone1970.tab files.

-- 
Cyrus Daboo

---

From: Ken Murchison <[email-masked]>
Date: Mon, 25 Aug 2014 11:56:53 -0400

On 08/25/2014 10:45 AM, Lester Caine wrote:
>> changedsince as currently defined can and will change if the 
>> iCalendar data of the resource changes, regardless of whether it 
>> changed because the underlying tzdata changed or because the 
>> translator being used constructed the iCalendar data differently. 
>> dtstamp and changedsince are only used by the client to fetch the 
>> latest version of iCalendar resources or a range of expansions. Its 
>> not designed for version history or anything along those lines. Its 
>> really the same as the HTTP Last-Modified response header, but with 
>> the advantage that when used with the list action it allows a client 
>> to tell the server to show it ALL resources which have changed since 
>> the given timestamp in one roundtrip rather than forcing the client 
>> to check all of the resources individually (each with its own 
>> roundtrip). 
> changedsince request against that dtstamp.

Correct, I was getting the impression that you were looking for 
timezones that changed between 2 dtstamps.

>   The un-answered question is

If you are talking about the expand action, then no, there would not be 
a response, per the current text in the draft:

       changedsince  OPTIONAL, but MUST occur only once.  If present, its
          value MUST be taken from the "dtstamp" result of a previous
          expand result.  If the targeted timezone has not changed over
          the expansion range queried in the request, then the server
          MUST return a 304 HTTP status response.

Now, having re-read that text, implementing this might be difficult for 
some implementations, including mine.  I'm thinking that this might need 
to be relaxed so that changedsince applies to ANY change in the timezone 
resource and not just changes that effect the expansion range in question.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 09:02:01 -0700

Cyrus Daboo wrote:

> I download the current tzdb from IANA (though the current test server
> is a few releases out of date)

Lester is referring to a file 'backzone' that is currently proposed for 
the tz data but has not been released.  'backzone' is a working area 
intended to hold zones that are out of scope for the tz database but 
that may be helpful as the basis of a hypothetical more-ambitious project.

In the current experimental version, 'backzone' data entries are not 
made available to clients.  And even assuming 'backzone' is published in 
the next tz release, I can't recommend its data for use in production, 
as its data entries are generally poorly-sourced and many entries are 
known to be wrong.  Finally, for technical reasons 'backzone' cannot be 
processed by 'zic' directly, so I assume it'd also take some massaging 
for your test server software to process it.

Unless you want to delve into such issues as provenance and 
data-reliability estimation, I wouldn't spend much time worrying about 
'backzone'.  All that being said, you can see the current experimental 
version of 'backzone' here:

https://github.com/eggert/tz/blob/master/backzone

---

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 17:17:37 +0100

> that may be helpful as the basis of a hypothetical more-ambitious project.

But it DOES contain information that has been well sourced and corrects
the early data on a number of timezones ...
Without this information, the basic tz data is not reliable pre 1970 so
usage of the truncated set should be identified!

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 25 Aug 2014 12:19:24 -0400

Hi Paul,

--On August 25, 2014 at 9:02:01 AM -0700 Paul Eggert <[email-masked]> 
wrote:

> intended to hold zones that are out of scope for the tz database but that
> may be helpful as the basis of a hypothetical more-ambitious project.
>
> In the current experimental version, 'backzone' data entries are not made
> available to clients.  And even assuming 'backzone' is published in the
> next tz release, I can't recommend its data for use in production, as its
> data entries are generally poorly-sourced and many entries are known to
> be wrong.  Finally, for technical reasons 'backzone' cannot be processed
> by 'zic' directly, so I assume it'd also take some massaging for your
> test server software to process it.
>

Ah, OK understood. Yes, right now my tool would ignore backzone if it 
appeared in the IANA download. That begs the question about what to do with 
publishers who offer "variants" of their time zone data (e.g., reliable 
post-1970 vs less reliable pre-1970 included).

So right now the tzdist protocol has a "primary-source" JSON object in its 
capabilities. That is just a free-form description of the source and it is 
up to the implementor to decide what to put in there (e.g., "Olson:2014f", 
"IANA-TZDB-2014f", "TZDB-2014g+backzone") etc.

So one question for the WG is whether we might need a more "formal" way to 
represent the source. Perhaps an IANA registry of "known" sources that 
provides a reliable set of expectations for validity range of the data? 
e.g., we could have:

urn:timezone:publisher:IANA
urn:timezone:publisher:Microsoft
urn:timezone:publisher:Foobar

With the option for "sub-namespace" elements like:

urn:timezone:publisher:IANA:post1970
urn:timezone:publisher:IANA:backzone

With those clearly stipulating the "variant" in use.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 17:25:25 +0100

> resource and not just changes that effect the expansion range in question.

list also has a changedsince option, which would be anticipated to give
a list limited by the truncated view being sourced? This does seem a
little strange without a means of only returning the timezones in the
format that created that listing? By the time the list has been read by
individual get's, it is quite possible that an entry may already have
been updated and so would be listed later while it has already been
supplied as the updated version.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 25 Aug 2014 18:38:11 +0200

Hi Cyrus,

On 8/25/14, 6:19 PM, Cyrus Daboo wrote:
>
> So one question for the WG is whether we might need a more "formal"
> way to represent the source. Perhaps an IANA registry of "known"
> sources that provides a reliable set of expectations for validity
> range of the data? e.g., we could have:
>
Certainly I was thinking something along these lines.
> urn:timezone:publisher:IANA:backzone
Errr.  I wasn't thinking THAT.  Two issues:

  * Presumably the sub-namespace has to be handled per publisher, and as
    such, we've just increased someone's workload.
  * There are some semantics tied in with those names- it's not just
    versioning.  I think that's asking for complexity, but maybe the
    complexity is already there?

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 12:34:56 -0700

>      complexity is already there?

Yes and no.  The tz database has several configuration options, but each 
has a default and the plain IANA version 2014f would surely just use the 
defaults.  Downstream publishers such as Apple and Debian often change 
the defaults (or even the data), but they're in different namespaces.

This raises a problem I have with Figure 1 of 
draft-douglass-timezone-service-11: its model of Contributors and 
Publishers doesn't match my understanding of how things work.  In 
practice the tz database acts as a sort of clearinghouse or "root 
publisher": downstream publishers like Debian republish modified 
versions and occasionally contribute patches back, and that's how the 
tz-mediated universe stays reasonably consistent even though minor 
details differ.  Figure 1 implies that contributors and publishers 
maintain independent databases, which isn't really what's going on.

---

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 20:42:02 +0100

>     complexity is already there?

The 'charter' for tz is only to support post 1970 data. Either we need
another publisher to take ownership of pre 1970 data, or we default to
showing the default IANA build as truncated at 1970 which is in fact the
situation. Then the backzone build can be used as un-truncated. I can
imagine that some servers may provide a different truncation point, so
while the sub-namespace may be useful to clarify the underlying source,
the truncation date should be enough for that.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 25 Aug 2014 15:46:08 -0400

Hi Paul,

--On August 25, 2014 at 12:34:56 PM -0700 Paul Eggert <[email-masked]> 
wrote:

> publisher": downstream publishers like Debian republish modified versions
> and occasionally contribute patches back, and that's how the tz-mediated
> universe stays reasonably consistent even though minor details differ.
> Figure 1 implies that contributors and publishers maintain independent
> databases, which isn't really what's going on.

Actually a "Contributor" does not maintain a database. Rather they are 
simply sources of information about time zone data. So this is how I would 
fit our diagram into the current IANA model:

1) Contributors are the members of the tz mailing list, or anyone else, 
that sends update requests to the tz co-ordinator.

2) The publisher is the tz co-ordinator/IANA.

3) Debian, Ubuntu (or anyone else shipping zoneinfo data in OS updates) is 
in effect a provider (currently not using tzdist).

What perhaps could be made clearer in the draft is that:

a) A provider may "tweak" the publisher's data for a variety of reasons - 
there are security implications here that we need to consider.

b) A contributor can potentially be anyone, including those fulfilling any 
of the other roles laid out in the architecture overview. e.g., Debian is a 
provider and also a contributor.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 12:53:11 -0700

Lester Caine wrote:
> it DOES contain information that has been well sourced and corrects
> the early data on a number of timezones

True, but these few nuggets can be hard to find among the mass of bad 
data.  For Isle of Man timestamps before 1883, 'backzone' is a few 
minutes better than the main-database alias to London.  But contrast 
this to most China timestamps before 1980, where 'backzone' can be hours 
worse than the main database.  'backzone' currently has so many 
China-sized problems that it should not be put into production use.

Perhaps 'backzone' could be annotated for reliability, so that people 
could more easily find its good nuggets and ignore its big mistakes. 
Such a project would be out of scope for tzdist, though.

> situation.

It's more complicated than that.  The tz database contains quite a bit 
of pre-1970 data to fill the pre-1970 eras of Zones that differ after 
1970.  Although some publishers omit this pre-1970 data, most keep it, 
and tzdist should support the typical configuration.

Using 'backzone' would be problematic in the tzdist environment for a 
reason that hasn't yet been discussed.  Until now, the tz database's 
configuration options have served to select subsets of the data: e.g., 
some publishers omit 'backward' and some keep it, but since 'backward' 
merely adds data the two sets of publishers won't disagree about data in 
both their scopes.  In contrast, if 'backzone' were used by some 
publishers, different  publishers would disagree about time stamps in 
the same Zone.  That would not be a good thing for users, and it's 
another reason that using 'backzone' would be problematic.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 13:06:10 -0700

Cyrus Daboo wrote:
> a) A provider may "tweak" the publisher's data for a variety of reasons
> - there are security implications here that we need to consider.
>
> b) A contributor can potentially be anyone, including those fulfilling
> any of the other roles laid out in the architecture overview. e.g.,
> Debian is a provider and also a contributor.

Sure, both those make sense.  I'd also make it clearer that the roles of 
republisher and root provider can and should overlap, in that a root 
provider should clearly identify its upstream data and should publish 
any tweaks made.  This is common practice in the tzdata distribution 
world, and it should help address some of the provenance issues that 
have been discussed on this list.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 25 Aug 2014 16:19:25 -0400

Hi Paul,

--On August 25, 2014 at 1:06:10 PM -0700 Paul Eggert <[email-masked]> 
wrote:

> provider should clearly identify its upstream data and should publish any
> tweaks made.  This is common practice in the tzdata distribution world,
> and it should help address some of the provenance issues that have been
> discussed on this list.

I don't think that information needs to be machine readable. So how about 
adding a "provider-details" object in the capabilities response that can 
contain a URI (probably http) to information about exactly what that 
providing is serving up.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Mon, 25 Aug 2014 21:22:30 +0100

> Such a project would be out of scope for tzdist, though.

Which is a different response to the same question on the tz mailing
list! :(

>From the point of view of the tzdist charter, tz is by default a
truncated database and publishing as 'untruncated' is the problem here.
It's content is limited and while that limitation may be earlier than
1970, that is the only date that is documented as the data being valid
from. Serving up data prior to 1970 is the area where there are
different levels of uncertainty and for some time zones it is known that
what is served up is incorrect either by deliberate omission, or because
the source data is suspect. For an 'untruncated' set of data from
tzdist, we either need an indication to the quality or in the first
iteration simply drop any attempt to support a clean untruncated server?
If someone can provide a publisher that has a better quality untruncated
set of data that we can use to test against, would be useful.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 16:25:02 -0700

Cyrus Daboo wrote:
> So how about adding a "provider-details" object in the capabilities
> response that can contain a URI (probably http) to information about
> exactly what that providing is serving up.

Yes, thanks, that sounds like a good idea.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 25 Aug 2014 16:47:27 -0700

Lester Caine wrote:
> from.

No, the tz documentation says that the database partitions the world 
into areas where clocks have agreed since 1970, which is a different 
thing.  For example, although 'America/Detroit' exists only because 
Michigan separately fiddled with its clocks in the early 1970s, the 
America/Detroit database entries go all the way back to the 19th century.

It's true that some pre-1970 data in the tz database is unsourced and 
suspect.  I'm trying to improve its quality, but any database of similar 
scope will have errors.  As there is no systematic representation of the 
reliability of the tz data entries, currently there's no good way to 
generate a quality indicator along the lines you suggest.

> set of data

There is no such publisher, unfortunately.  Perhaps something better 
will come along eventually, but for now we can't ask the tzdist protocol 
to supply information that doesn't exist.

---

From: Tobias Conradi <[email-masked]>
Date: Tue, 26 Aug 2014 01:57:02 +0200

On Tue, Aug 26, 2014 at 1:47 AM, Paul Eggert <[email-masked]> wrote:
> No, the tz documentation says that the database partitions the world into
> areas where clocks have agreed since 1970, which is a different thing.
What the tz documentation says is irrelevant. There is enough evidence
that actual data deviates.

Can the chairs please intervene?

> suspect.  I'm trying to improve its quality,
I think what individuals try on their databases is irrelevant to the
scope of this WG.

> but any database of similar
> scope will have errors.
But maybe less.

> There is no such publisher, unfortunately.
This is an unprovable statement. I think it is also out of scope of the WG.

>  Perhaps something better will
> come along eventually, but for now we can't ask the tzdist protocol to
> supply information that doesn't exist.
That is not true. One can always ask. To obtain an answer is not a
requirement for asking first.

I find such statement highly disruptive, can the chairs please intervene?

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 08:07:17 +0100

> generate a quality indicator along the lines you suggest.

Paul - you know what my complaint is, and backzone contains well sourced
information which we have researched, but is being blocked from being
published in the main tz database BECAUSE of the 1970's rule. Your
recent patch on asia originally said - ignoring because pre 1970 cutoff
- when in fact that data simply has insufficient detail to confirm what
SHOULD replace the current data. My own work on the UK was submitted in
2012 and further research was obviously pointless because as at that
time the work was not accepted. I'd not spotted that while the work has
now been included in backzone it has not ACTUALLY been published yet.

In the context of THIS WG, that pre-1970 data exists in tz that is not
being published needs to have some mechanism to identify the fact. Some
timezones that are being published on the current demo are historically
wrong and there is no mechanism to indicate that is the case other than
flagging as truncated! If the current draft is to move forward I propose
that nothing is currently published as 'un-truncated' until such times
as the 'hypothetical project' using all available data which can also
identify where it's accuracy may be suspect is available as a source. We
should not be producing a service which is KNOWN to be publishing
misleading information? Better not to publish anything so we know we
have to look deeper for the facts about that data?

---

From: Eliot Lear <[email-masked]>
Date: Tue, 26 Aug 2014 09:34:53 +0200

On 8/26/14, 9:07 AM, Lester Caine wrote:
> being published needs to have some mechanism to identify the fact.

Why?  Why isn't that happening above the layer of the protocol?  The
client connects to the source and those who coded up the client ought to
have some understanding of the nature of the underlying data that is
returned.  Having the ability to truncate the data should be sufficient
to answer your concern.  Put another way, we are not going to quality
control the data through this protocol.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 09:08:32 +0100

> control the data through this protocol.

At this stage as long as the data IS truncated I would be happy. The
current demo is not tagged as truncated, but then I doubt that Cyrus was
even aware of the problem with the data - which is exactly why this is
so important!. If the protocol is not going to carry any 'quality'
information then data that is not fully validated is simply truncated?

There MAY be a case for truncating different timezones to different
dates, which was the discussion on a 'start date', but the current demo
is not producing historically correct start dates currently anyway.

---

From: Paul Eggert <[email-masked]>
Date: Tue, 26 Aug 2014 01:21:06 -0700

Lester Caine wrote:
> Some timezones that are being published on the current demo are historically wrong

If I understand this correctly, you're saying not that the time zones 
are wrong, but that the aliases are wrong.  For example, 
Europe/Isle_of_Man is an alias for Europe/London, and you're saying that 
this is misleading for 19th-century time stamps because the Isle of Man 
and London were at different GMT offsets back then.  Similarly, 
Asia/Istanbul is an alias for Europe/Istanbul, and this also could be 
considered misleading for old time stamps because districts of Istanbul 
were at slightly different GMT offsets way back when.

In this sense, most of the aliases in the tz database are "wrong". 
After all, London is not the Isle of Man, and Tijuana is not Ensenada, 
and European Istanbul is not Asian Istanbul.  But right or wrong the 
aliases exist in the tz database, mostly for backward compatibility 
reasons, and they're not likely to go away any time soon.

 From the tzdist point of view I suppose the simplest way to fix these 
"wrong" aliases would be to omit them, but that sounds a bit drastic....

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 09:42:52 +0100

On 26/08/14 09:21, Paul Eggert wrote:
> In this sense, most of the aliases in the tz database are "wrong". After
> all, London is not the Isle of Man, and Tijuana is not Ensenada, and
> European Istanbul is not Asian Istanbul.  But right or wrong the aliases
> exist in the tz database, mostly for backward compatibility reasons, and
> they're not likely to go away any time soon.
> 

It's a bit of a circular argument ... Do you not agree that the use of
GMT started on different dates even in different parts of the UK?
Pretending they all started on the one date is what is wrong here. And
as yet I'm not 100% certain later changes all happened on the same day
as London ... but I've not gone back through all the source documents
which are all now published. Using the London data as if that is a fact
for all the other UK timezones is what is wrong.

It is the same with the material that is now being uncovered in other
parts of the world which confirm other facts, which may need a current
tz identity to be split. This is happening where counties in the states
are moving their DST choices today, and back in the past we know things
started on different dates, but perhaps currently we do not have proof
of the exact date? We can however say that we know that they did not
happen on the same date so using one set of data or all is known to be
wrong?

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 11:22:46 +0100

> "wrong" aliases would be to omit them, but that sounds a bit drastic....

Actually the fact is that providing a truncated view which only has
tzid's for timezones valid for that view is probably all I'm asking for
here. We have a much bigger set of tzid's not only for backwards
compatibility, but also because the base data IS truncated. That tzdist
then refuses to supply Europe/Isle_of_Man because it is not available
and one has to choose an alternative FROM THAT SERVER is probably the
correct process. It passes the problem to the lookup and removes the
incorrect information being served up? But the truncation is still not
to a particular date ... so tagging it in capabilities becomes the problem?

---

From: Paul Eggert <[email-masked]>
Date: Tue, 26 Aug 2014 10:00:59 -0700

Lester Caine wrote:
> Do you not agree that the use of

Sure, but in this sense even the zone Europe/London is "wrong" or 
"truncated", as it could give users the mistaken impression that all of 
London switched to GMT in December 1847, which is not what actually 
happened.  And in the same sense almost all tz zones are "wrong" or 
"truncated", as they give a too-tidy summary of the introduction of 
standard time.

Fixing this "wrongness" would be a big job, though.  I expect it'd 
require a tz database schema change and some tzcode API changes, in 
order to represent our knowledge about (and ignorance of) long-ago 
timekeeping, and this would be outside tzdist's scope.

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 18:48:58 +0100

> timekeeping, and this would be outside tzdist's scope.

Whether people actually changed their clocks when a new 'rule' came into
force is not something we can ever cater for? I know there are examples
everywhere of local communities not bothering with their governments
decrees, adoption of DST being a prime example. However if we have
statutory notices providing fixed points in the calender of events then
these should be recognised in the base data?

In the light of current discussions on formats, the fact that the bulk
of what I am talking about is perhaps a few extra historic dates
pre-dating a more common set of data, the ability to retain the common
rule sets that form the base of tz data makes a lot of sense. We only
need a small number of additional rules, rather than having to duplicate
the whole Europe/London VTIMEZONE dataset for each of the extra tzid's :(

( Tobias - This is part of what I'm struggling with with iCalendar
conversions. The database would be a lot smaller if the common rule sets
could be reused when defining many of the tzid sequences. If a modern
DST rule changes every tzid has to be re-download completely when all
that has changed is on small area of common data )

---

From: Paul Eggert <[email-masked]>
Date: Tue, 26 Aug 2014 11:17:25 -0700

Lester Caine wrote:
> force is not something we can ever cater for?

That issue is outside the tzdist scope now, yes.  Although maybe some 
future expanded version of the tz database could address it and tzdist 
could then pick it up, I expect this would require a change to the 
tzdata scheme, to the tzcode API, to VTIMEZONE, etc., and this is not 
something we can realistically put into the protocol now.

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 19:25:35 +0100

> something we can realistically put into the protocol now.

Which still sidesteps the data that IS fully documented but not allowed
to be used in a default build of tz data because of the 1970 rule :(

---

From: Eliot Lear <[email-masked]>
Date: Tue, 26 Aug 2014 20:31:58 +0200

> something we can realistically put into the protocol now.

Although as a general design goal, some limited extensibility should be
present.  I think perhaps there ought to be an issue on that.

Eliot

---

From: Tim Parenti <[email-masked]>
Date: Tue, 26 Aug 2014 15:49:53 -0400

> present.  I think perhaps there ought to be an issue on that.

Agreed.  I had a thing mostly written up, but most of it is less relevant
now.  In any case, I don't think its unreasonable to presume that any given
tzdist publisher might have reasons to provide more than one "set" of (for
lack of a better phrase) mutually-incompatible data, and that clients might
have reasons to select any of these.  We'd want to take care to explain
what is meant by that; as in, something distinct from merely versions of
the same dataset, so as not to exacerbate the distribution issues that
tzdist is aiming to address in the first place.  For example, for tz, this
could be the standard distribution, and a backzone-extended distribution.

tzdist should remain mindful of the work that's being done in this space
(both by tz and by others) and at least consider the case that such data,
and the desire to distribute it, might exist someday soon.  Right now, our
implementation costs are relatively low.  The "sets" available would be
added to capabilities, alongside truncation information.  Namespacing would
be an issue, but as most package providers today already handle this in
some way already, I don't think it's a deal-breaker here.

If this is a concrete-enough proposal to create a ticket in trac, I think
it would address the main parts of Lester's concern here, and I would
support that.  (As much of the rest of this discussion has been outside our
scope.)

--
Tim Parenti

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 21:13:15 +0100

On 26/08/14 20:49, Tim Parenti wrote:
> If this is a concrete-enough proposal to create a ticket in trac, I
> think it would address the main parts of Lester's concern here, and I
> would support that.  (As much of the rest of this discussion has been
> outside our scope.)

I am very conscious of the fact that a lot of this is outside the scope,
but I think most of you now appreciate that simply ignoring the
complexity of the underlying data may result in problems later on? In
reality I don't think the data niggles are show stoppers from the
current draft and it is just a matter of identifying the right hooks to
use to be consistent across data sources?

---

From: Tim Parenti <[email-masked]>
Date: Tue, 26 Aug 2014 16:30:20 -0400

On 26 August 2014 16:13, Lester Caine <[email-masked]> wrote:

> I think most of you now appreciate that simply ignoring the
> complexity of the underlying data may result in problems later on
>

Agreed.  Neither project exists in a silo.  After all, if all goes to plan,
tz will be one of the largest providers to tzdist, and tzdist one of the
largest consumers of tz.  I thank you for your forward thinking here.

For the purposes of turning this into an actionable ticket for tzdist,
would you say that my proposal at
http://www.ietf.org/mail-archive/web/tzdist/current/msg00222.html regarding
supporting multiple datasets per publisher would address your concern?

--
Tim Parenti

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 26 Aug 2014 16:34:43 -0400

Hi Lester,

> use to be consistent across data sources?

So one question is how much of that "meta-data" needs to be included in the 
time zone data itself, vs be "accessible" via the protocol through some 
action. The specific use case I am thinking of is the geo->time zone 
mapping. In the general case one might want to have some form of geospatial 
mapping (e.g., KML) describe the regions where a particular time zone is 
applicable. But should that data be in the time zone itself? Is it 
something the clients actually want in the "raw" data? Perhaps all the 
clients want is a service to map a geo co-ordinate to a time zone, and 
tzdist could easily be extended to provide such an action.

That said, lets say in the future iCalendar were extended to support a 
property to hold the KML data and historical comments from the TZ DB etc. 
We could extend tzdist to use a "filter" query param with various tokens to 
indicate what "parts" of the TZ data should be returned in the VTIMEZONE 
object. The default would be to not return the new properties, but a client 
that did want the KML, could use "&filter=kml" in its request and the 
server would include the KML properties. That is not hard for us CalDAV 
folks to do as CalDAV already has a filtering mechanism like that.

The main point I want to make here is that we have allowed multiple 
extension points in the protocol. Specifically one can add new actions and 
new query parameters. The capabilities mechanism exposes exactly what 
actions and parameters the sever supports, and we have registries for those 
to be standardized for interoperability.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Tue, 26 Aug 2014 14:02:36 -0700

Cyrus Daboo wrote:
> Perhaps all the clients want is a service to map a geo co-ordinate to a
> time zone, and tzdist could easily be extended to provide such an action.

This sounds better than trying to stuff the geospatial data into the 
zone itself.  We're talking about a part of the draft I haven't read 
yet, but let me try to indicate the scope of the problem so that you can 
think about whether the protocols' extension points will handle it.

Filtering the tz database will entail changes to the corresponding 
geospatial mappings.  For example, suppose a provider excludes data 
before the year 2000 (either for all queries, or for just the current 
query).  Then America/Detroit can become an alias for America/New_York, 
even though it's not an alias in the unfiltered database because they 
differed in the early 1970s.  Hence the geospatial data corresponding to 
America/Detroit will differ in the filtered database.  And even the 
geospatial data for America/New_York will differ, because with some data 
filters America/New_York will cover downstate Michigan and with others 
it won't.

A similar issue occurs if one grows the database by adding 'backzone' or 
suchlike data: this will change the geospatial data for existing zones, 
as well as requiring more geospatial data for the new zones.

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 22:18:20 +0100

On 26/08/14 21:34, Cyrus Daboo wrote:
> Hi Lester,
> 
> --On August 26, 2014 at 9:13:15 PM +0100 Lester Caine
> <[email-masked]> wrote:
> 
> So one question is how much of that "meta-data" needs to be included in
> the time zone data itself, vs be "accessible" via the protocol through
> some action. The specific use case I am thinking of is the geo->time
> zone mapping. In the general case one might want to have some form of
> geospatial mapping (e.g., KML) describe the regions where a particular
> time zone is applicable. But should that data be in the time zone
> itself? Is it something the clients actually want in the "raw" data?

geonames privides a very usable lookup for converting a geolocation to a
timezone, and is currently using the full 'zone.tab' list from tz. But
it has been discussed that we may need to provide an additional layer if
substantial material comes to light prior to 'conventional GMT'
timezones. Paul is worried that this would create hundreds of new id's,
but I tend to the view that there would be an historic lookup service of
some sort. THAT is where the importance of the common rules sets from tz
become important and provide the bulk of the data, but it may be
possible to use iCalendar records instead.

> property to hold the KML data and historical comments from the TZ DB
> etc. We could extend tzdist to use a "filter" query param with various
> tokens to indicate what "parts" of the TZ data should be returned in the
> VTIMEZONE object. The default would be to not return the new properties,
> but a client that did want the KML, could use "&filter=kml" in its
> request and the server would include the KML properties. That is not
> hard for us CalDAV folks to do as CalDAV already has a filtering
> mechanism like that.

I'd flagged elsewhere that while the rule section of VTIMEZONE has a
comment element, it seems to be missing from the header block? We can
add the rule source comments ( many times over for common rules ;) ) but
can't easily include the timezone comment body? I did concider using the
'initial' rule to hold the main comment body, but on some this is a part
of a DST pair rather than a simple 'started' rule ...

> extension points in the protocol. Specifically one can add new actions
> and new query parameters. The capabilities mechanism exposes exactly
> what actions and parameters the sever supports, and we have registries
> for those to be standardized for interoperability.

The database is all set up for 'find' to return a lookup on the geonames
database :)
Just need to interface it with the tzdist code ...

---

From: Lester Caine <[email-masked]>
Date: Tue, 26 Aug 2014 22:53:10 +0100

On 26/08/14 21:30, Tim Parenti wrote:
> http://www.ietf.org/mail-archive/web/tzdist/current/msg00222.html
> regarding supporting multiple datasets per publisher would address your
> concern?

Certainly any restrictions being applied to the underlying data set
outside of simply truncating it should be identified, but I would expect
the publisher to make that information available via a link to their
data source website? I'm not sure that a service would want to handle
any more than one data set at a time? So I think I would expect
different URL's for different types of service? So while it would be
useful to know that other sets are available haven't we already made a
choice when asking for a capability message so we are just checking that
we have the right source?

---

From: Tim Parenti <[email-masked]>
Date: Tue, 26 Aug 2014 22:42:23 -0400

> we have the right source?

Ah, yes, separate URIs for tzdist access points makes sense and avoids
complicating the protocol.  I don't know what was preventing me from seeing
that earlier.

I think it might be reasonable to put some sort of sanity-check like this
in a response.  Is this what you're asking for?

--
Tim Parenti

---

From: Lester Caine <[email-masked]>
Date: Wed, 27 Aug 2014 07:32:41 +0100

On 27/08/14 03:42, Tim Parenti wrote:
> complicating the protocol.  I don't know what was preventing me from
> seeing that earlier.
> 
> I think it might be reasonable to put some sort of sanity-check like
> this in a response.  Is this what you're asking for?

Yes.
Since WHAT is being supplied is not within the scope of the WG, all we
can ask for at this stage is a 'sanity-check'?
Any more than that would need a change to the charter?

4.2.1 Discovery either knows exactly which URL to use or has an
undefined 'discover it through some mechanism'

What I am trying to understand here is if the '/stdtimezones' element of
the demo server URL is a point where one might have alternate views such
as '/fulltimezones' for an untruncated view while the stdtimezones view
is a simply post 1970 truncated view? So the request-URI could
potentially be a mechanism to add a '/sources' lookup?

---

From: Tim Parenti <[email-masked]>
Date: Thu, 28 Aug 2014 00:09:23 -0400

> can ask for at this stage is a 'sanity-check'?

Yes, I think it would be useful for responses to contain information about
where they came from.

On 27 August 2014 02:32, Lester Caine <[email-masked]> wrote:

> So the request-URI could

Yes, I think this could be appropriate.  It's sort of edge-of-scope,
though, as we already mentioned that, presumably, by accessing a particular
tzdist endpoint's URI we have made an informed choice about the dataset we
want.

I think this is the only active thread not yet in trac, so I'm trying to
distill this down into potentially actionable items.

--
Tim Parenti