Subject: [Tzdist] Semantics of year fields, and clarifying onset

From: Tim Parenti <[email-masked]>
Date: Sat, 23 Aug 2014 00:58:35 -0400

In Asia/Singapore, there is a transition from 1982-01-01T00:00+07:30 to
1982-01-01T00:30+08:00.
https://github.com/eggert/tz/blob/3962fae49c5294f0ce79353632a93b5a1db206f6/asia#L2468
However, this transition can also be said to have taken place at
1981-12-31T16:30Z.  So, for the purposes of data truncation and start/end
fields in expansion requests, in which year is this transition considered
to have happened?  1981 or 1982?

In other words, while it's fairly clear to me that the year fields refer to
01-01T00:00 as the reference points for each boundary (although maybe that
should be made clearer in the draft), is that to be interpreted as local or
UTC?  If the former, which local?

Currently, this demo includes it in 1982...
https://demo.calendarserver.org:8443/stdtimezones?action=expand&tzid=Asia/Singapore&start=1980&end=1983
{
  "dtstamp":"2014-06-09T19:47:04Z",
  "observances":[
    {
      "name":"SGT",
      "onset":"1980-01-01T00:00:00",
      "utc-offset-to":27000,
      "utc-offset-from":27000
    },
    {
      "name":"SGT",
      "onset":"1982-01-01T00:00:00",
      "utc-offset-to":28800,
      "utc-offset-from":27000
    }
  ]
}

...but not in 1981...
https://demo.calendarserver.org:8443/stdtimezones?action=expand&tzid=Asia/Singapore&start=1980&end=1982
{
  "dtstamp":"2014-06-09T19:47:04Z",
  "observances":[
    {
      "name":"(SGT)",
      "onset":"1980-01-01T00:00:00",
      "utc-offset-to":27000,
      "utc-offset-from":27000
    }
  ]
}

...so that would imply that this demo is currently treating it as local.

I can see either practice being valid, but I think it might be wise to
specify one (preferably UTC) in the protocol.  Additionally, I think it
would help, even just with readability, to have either offset information
specified directly in the onset field, or the onset field specified in UTC.

Alternatively, we could require that implementations include the full UTC
timestamps used for cutoffs within their responses, to ensure that clients
know for exactly what set of timestamps they have received complete
information.  For a server using 01-01T00:00 UTC as cutoffs, this could
look something like:

{
    "dtstamp": "2014-06-09T19:47:04Z",
    "observances": [
        {
            "name": "SGT",
            "onset": "1980-01-01T07:30:00+0730",
            "utc-offset-from": 27000,
            "utc-offset-to": 27000
        },
        {
            "name": "SGT",
            "onset": "1982-01-01T00:00:00+0730",
            "utc-offset-from": 27000,
            "utc-offset-to": 28800
        }
    ],
    "observance-bounds": [
        "1980-01-01T07:30:00+0730",
        "1982-01-01T08:00:00+0800"
    ]
}

I think the extra data required in this version is perhaps a bit much, so I
would propose simply specifying the semantics of these year fields in the
protocol.

--
Tim Parenti

---

From: Paul Eggert <[email-masked]>
Date: Fri, 22 Aug 2014 23:44:24 -0700

Tim Parenti wrote:
> specify one (preferably UTC) in the protocol.

It should be Universal Time, to avoid ambiguity if there are two 
distinct Universal Time stamps that both correspond to January 1 at 
00:00:00 local time.  This can happen if the local clock is set back 
shortly after midnight on January 1, something that's rare but is not 
unheard of.  One example is the America/Phoenix zone, where the clock 
jumped backward from 1944-01-01 00:01 to 1943-12-31 23:01.

A related ambiguity is the use of the word "midnight", which could be 
00:00 and could be 24:00.

To fix both ambiguities, in section 3.4 I propose changing "start of a 
year, i.e., the year's first midnight for the timezone's local time" to 
"start of a year, i.e., 00:00 UTC January 1 in that year", and in 
Section 6.4 I propose changing "(midnight local time, January 1st of the 
start year)" to "(00:00 UTC, January 1st of the start year)".

---

From: Tobias Conradi <[email-masked]>
Date: Sat, 23 Aug 2014 14:52:00 +0200

On Sat, Aug 23, 2014 at 6:58 AM, Tim Parenti <[email-masked]> wrote:
> fields in expansion requests, in which year is this transition considered to
> have happened?  1981 or 1982?
Both.

> should be made clearer in the draft)
Would that mean a redefinition of a topic covered in ISO 8601?

>, is that to be interpreted as local or
If not specified in the request it could be either.

> I can see either practice being valid,
Especially since neither is exclusively correct.

>but I think it might be wise to
That would restrict the protocol to support only a subset of reality.

> information.  For a server using 01-01T00:00 UTC as cutoffs, this could look
> something like:
>
> {
Not a valid ISO 8601 representation.

>             "onset": "1982-01-01T00:00:00+0730",
Not a valid ISO 8601 representation.

>         "1982-01-01T08:00:00+0800"
Not valid ISO 8601 representations.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Steve Allen <[email-masked]>
Date: Sat, 23 Aug 2014 08:29:41 -0700

On Fri 2014-08-22T23:44:24 -0700, Paul Eggert hath writ:
> 00:00:00 local time.

If this is adopted it might be good to point out in the draft that on
platforms where the system clock is aware of leap seconds this implies
a need for cautious coding around year boundaries that contain a leap
second.  Although tzdist is aimed at calendaring applications which
have (appropriately, IMHO) declined to consider leap seconds, the
programmer will have to take care with the handling of Dec 31 23:59:60
and Jan 1 00:00:00.

> To fix both ambiguities, in section 3.4 I propose changing "start of
> a year, i.e., the year's first midnight for the timezone's local
> time" to "start of a year, i.e., 00:00 UTC January 1 in that year",
> and in Section 6.4 I propose changing "(midnight local time, January
> 1st of the start year)" to "(00:00 UTC, January 1st of the start
> year)".

Phrasing this as UTC fails for 1960 Jan 1 and any year boundary before
that.  The agreement to "coordinate" time happened in 1959 August and
was aimed to take place in 1960, but the time bureaus did not get the
coordination scheme operational until some date later in the year.
For 1960-01-01 and before the term UTC cannot apply to any
contemporary notion of what time it was.  If the idea is to avoid
ambiguity then for 1960 and before this requirement must be phrased as
"UT", not "UTC".

--
Steve Allen                 <[email-masked]>                WGS-84 (GPS)
UCO/Lick Observatory--ISB   Natural Sciences II, Room 165    Lat  +36.99855
1156 High Street            Voice: +1 831 459 3046           Lng -122.06015
Santa Cruz, CA 95064        http://www.ucolick.org/~sla/     Hgt +250 m

---

From: Tobias Conradi <[email-masked]>
Date: Sat, 23 Aug 2014 19:11:03 +0200

On Sat, Aug 23, 2014 at 5:29 PM, Steve Allen <[email-masked]> wrote:
> Although tzdist is aimed at calendaring applications which
> have (appropriately, IMHO) declined to consider leap seconds,
"Consideration" of leap seconds is required to parse time stamps
ending in "Z" in accordance with ISO 8601.

> contemporary notion of what time it was.
What is "contemporary notion"? Is there "contemporary notion" for
future time points?

> If the idea is to avoid
Does that mean
- UT is less ambiguous than UTC
- one cannot convert between the two
?

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Steve Allen <[email-masked]>
Date: Sat, 23 Aug 2014 10:36:38 -0700

> What is "contemporary notion"? Is there "contemporary notion" for

The future is always a guess that is subject to revision when
some bureaucrat decides everyone should reset their clocks.

The aim of the tz project has been to record what people actually do.
At the beginning of 1960 and before nobody could have been using UTC,
so any use of that term implies some kind of extrapolation for which
there may be different interpretations.  (Those might include
extrapolating some kind of uniform/atomic time scale, which is not
appropriate to do for time stamps of any civil activities.)

The notion of UT using those words existed from 1884.  Even though
different agencies used different algorithms until 1901, to within the
accuracy of recorded civil time stamps, any algorithm for the concept
of UT is sufficiently valid for this project throughout all of
history.

> - one cannot convert between the two

Starting from the formation of the Bureau International de l'Heure the
differences between the time scales provided by various time bureaus
is recorded in the issues of Bulletin Horaire.  In much the same way
as the history of time zones is both complex and incomplete, so are
the records of which national clocks were running fast or slow by how
many milliseconds.  Those differences are the realm of specialist
applications, and they are outside the scope of this calendaring
project.

---

From: Tobias Conradi <[email-masked]>
Date: Sat, 23 Aug 2014 20:31:10 +0200

> applications, and they are outside the scope of this calendaring
If the last sentence is true, I suggest to not replace UTC which is
used in ISO 8601 with something that is only related via a mechanism
that is "outside the scope", i.e. I suggest to not apply

>>Steve Allen:

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Paul Eggert <[email-masked]>
Date: Sat, 23 Aug 2014 13:11:27 -0700

> ambiguity then for 1960 and before this requirement must be phrased as

Sure, that should be easy enough, with the following proposed changes to 
the draft:

* In Section 1, change "coordinated universal time (UTC)" to "Universal 
Time (UT)".

* In Section 9, change "local or UTC time" to "local time or Universal 
Time".

* Insert the following definition in section 1.2's glossary: "Universal 
Time (UT): the time standard based on the Earth's rotation, where 
Coordinated Universal Time (UTC) is assumed for UT time stamps since UTC 
was introduced in 1960."

* Replace all other instances of "UTC" with "UT".

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Sat, 23 Aug 2014 21:40:11 +0100

> Coordinated Universal Time (UTC) is assumed for UT time stamps since UTC 

I disagree.

UT and UTC are not the same thing. Confusing them in this way will just add
to the general confusion about this subject.

-- 
Clive D.W. Feather          | If you lie to the compiler,
Email: [email-masked]     | it will get its revenge.
Web: http://www.davros.org  |   - Henry Spencer
Mobile: +44 7973 377646

---

From: Tobias Conradi <[email-masked]>
Date: Sat, 23 Aug 2014 22:46:49 +0200

On Sat, Aug 23, 2014 at 10:11 PM, Paul Eggert <[email-masked]> wrote:
> Sure, that should be easy enough, with the following proposed changes to the
> draft:
>
> * In Section 1, change "coordinated universal time (UTC)" to "Universal Time
> (UT)".
>
> * Insert the following definition in section 1.2's glossary: "Universal Time
> (UT): the time standard based on the Earth's rotation, where Coordinated
> Universal Time (UTC) is assumed for UT time stamps since UTC was introduced
> in 1960."
>

Using a term for which according to Steve Allen the transformation
into UTC is out of scope:

>Those differences are the realm of specialist
avoids ambiguity?

Uncertainty about what time might be meant to avoid ambiguity?

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Paul Eggert <[email-masked]>
Date: Sat, 23 Aug 2014 14:34:28 -0700

Clive D.W. Feather wrote:
> UT and UTC are not the same thing.

The proposed wording attempts to outline the situation without 
descending into eyeroll-inducing minutiae that is out of scope.  I was 
taught that UT is a family of timescales that includes UT1, UTC, etc. 
If you were taught at a different school please feel free to propose 
improved wording, but let's hope the draft doesn't have to go down the 
GMT/UT0/UT1/UT2/... rabbit-hole.

---

From: Tobias Conradi <[email-masked]>
Date: Sun, 24 Aug 2014 06:22:46 +0200

On Sat, Aug 23, 2014 at 11:34 PM, Paul Eggert <[email-masked]> wrote:
> The proposed wording attempts to outline the situation without descending
> into eyeroll-inducing minutiae that is out of scope.  I was taught that UT
> is a family of timescales that includes UT1, UTC, etc. If you were taught at
> a different school please feel free to propose improved wording, but let's
> hope the draft doesn't have to go down the GMT/UT0/UT1/UT2/... rabbit-hole.
By replacing UTC with UT you pushed it there.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Tim Parenti <[email-masked]>
Date: Sun, 24 Aug 2014 22:53:30 -0400

> * Replace all other instances of "UTC" with "UT".

Thanks for finding those, Paul.  Additionally, there several references to
"January 1st" in section 6.4 of
http://tools.ietf.org/html/draft-douglass-timezone-service-11, which should
probably all be similarly clarified.  Also, I just noticed that 6.4 refers
to "January 1st" while 3.4 refers to "1st January"; the "st" is superfluous
in either case and should probably be removed, but let's at least pick one
order or the other.

--
Tim Parenti

---

From: Tobias Conradi <[email-masked]>
Date: Mon, 25 Aug 2014 05:23:42 +0200

> in either case and should probably be removed, but let's at least pick one

Clarify "UTC" by replacing it with "UT", when Paul just said "I was
taught that UT is a family of timescales that includes UT1, UTC, etc."

I vote this for being included in a Monty Python video in case they
make a new one.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Paul Eggert <[email-masked]>
Date: Wed, 27 Aug 2014 18:04:44 -0700

Re Ticket #21 <http://trac.tools.ietf.org/wg/tzdist/trac/ticket/21>, Tim 
Parenti wrote:

> Thanks for finding those, Paul.  Additionally, there several references to "January 1st" in section 6.4 of http://tools.ietf.org/html/draft-douglass-timezone-service-11, which should probably all be similarly clarified.

Sounds good.  Also, RFC3339 already covers the issue of pre-UTC 
timestamps, so let's just defer to that.  So here's a revised proposal 
for changing the wording, this time relative to 
draft-ietf-tzdist-service-00:

* Insert the following as a new section between 3.3 and 3.4: "3.x. 
Universal Time.  Timestamps can occur before the introduction of UTC. 
As per [RFC3339] section 1, such timestamps are expressed relative to 
universal time, using the best available practice at the stated time."

* In section 3.4 change "start of a year, i.e., the year's first 
midnight for the timezone's local time" to "start of a year, i.e., 00:00 
UTC January 1 in that year".

* In Section 6.4 change "January 1st of that year" (twice) and "midnight 
local time, January 1st of the start year" (once) to "00:00 UTC, January 
1 of that year" (all 3 occurrences).

* In Section 9, change "local or UTC time" to "local time or UTC".

---

From: Steve Allen <[email-masked]>
Date: Wed, 27 Aug 2014 19:03:02 -0700

On Wed 2014-08-27T18:04:44 -0700, Paul Eggert hath writ:
> Also, RFC3339 already covers the issue of pre-UTC
> timestamps, so let's just defer to that.

> Universal Time.  Timestamps can occur before the introduction of
> UTC. As per [RFC3339] section 1, such timestamps are expressed
> relative to universal time, using the best available practice at the
> stated time."

Thanks, that will be as good as is needed for the purposes of this
protocol, and its meaning is adequately well defined even for events
prior to the establishment of the Greenwich Observatory.  (Trying to
encode the content of Bulletin Horaire and its analyses of the scores
of different sources and several meanings of time which were used
would be ridiculously beyond the scope of this protocol.)

---

From: Tim Parenti <[email-masked]>
Date: Wed, 27 Aug 2014 23:33:00 -0400

Thanks, Paul; I think this is good.  I would be consistent, however, about
using a comma after "00:00 UTC" in 3.4 and 6.4.  For the sake of a concrete
suggestion, I don't think a comma is necessary here.  Likewise, the
preposition after "January 1" should be "of".

As for the second part of #21, about "includ[ing] the full UTC timestamps
used for cutoffs within...responses", I think this is more-or-less covered
by the second paragraph of
http://trac.tools.ietf.org/wg/tzdist/trac/ticket/15.  If I recall
correctly, the test servers don't currently implement this, but I think
it's important to include this information in the response somehow, so
clients know exactly what subset of data they're getting, and can verify
that it is as expected.

--
Tim Parenti

On 27 August 2014 21:04, Paul Eggert <[email-masked]> wrote:

>  Thanks for finding those, Paul.  Additionally, there several references
>> to "January 1st" in section 6.4 of http://tools.ietf.org/html/
>> draft-douglass-timezone-service-11, which should probably all be
>> similarly clarified.
>>
>
> timestamps, so let's just defer to that.  So here's a revised proposal for
> changing the wording, this time relative to draft-ietf-tzdist-service-00:
>
> Universal Time.  Timestamps can occur before the introduction of UTC. As
> per [RFC3339] section 1, such timestamps are expressed relative to
> * In section 3.4 change "start of a year, i.e., the year's first midnight
> for the timezone's local time" to "start of a year, i.e., 00:00 UTC January
> 1 in that year".
>
> local time, January 1st of the start year" (once) to "00:00 UTC, January 1
> of that year" (all 3 occurrences).
>

---

From: Paul Eggert <[email-masked]>
Date: Wed, 27 Aug 2014 22:52:58 -0700

> clients know exactly what subset of data they're getting, and can verify

I'm afraid it's not, because of another problem I just noticed.  Suppose 
the server is based on POSIX 32-bit signed time_t, and so can represent 
time stamps in the range 1901-12-13 20:45:52 through 2038-01-19 03:14:07 
UTC.  The protocol would force this server to advertise the year range 
[1902,2038), thus arbitrarily discarding some timestamps in 1901 and 
some in 2038.

The obvious way to fix this is to change truncation times to allow 
1-second precision, instead of the current 1-year precision.  That is, 
section 5.4, 5.5 and 5.9 would use UTC timestamps, not year numbers. 
This was proposed near the end of Ticket 21, and so it appears to be 
good idea for multiple reasons.  However, unlike Ticket 21, I expect the 
observance bounds should be in UTC, not in local time, as this would 
avoid local-time weirdnesses.

> I would be consistent, however, about

Yes, thanks, that all sounds good.

---

From: Tim Parenti <[email-masked]>
Date: Thu, 28 Aug 2014 13:20:28 -0400

On 28 August 2014 01:52, Paul Eggert <[email-masked]> wrote:

> The protocol would force this server to advertise the year range
> [1902,2038), thus arbitrarily discarding some timestamps in 1901 and some
> in 2038.

I wouldn't be too upset by such a compromise.  Perhaps clients could see
this pattern and try to intuit that a slight bit more information might be
available.  Clients could request a wider range such as [1901,2039), as
long as the server's response clearly states that the validity of its
response is actually bounded by [1901-12-13T20:45:52Z,
2038-01-19T03:14:08Z).  Being more precise about the advertised end date
only buys upstream data providers an extra 18 days and change to update
their systems to handle 64-bit timestamps, which for calendaring purposes
they should have hopefully long since had in place by then.

On 28 August 2014 01:52, Paul Eggert <[email-masked]> wrote:

> section 5.4, 5.5 and 5.9 would use UTC timestamps, not year numbers. This
> was proposed near the end of Ticket 21, and so it appears to be good idea
> for multiple reasons.

To clarify, I didn't propose allowing second-level precision truncation
boundaries, only that the boundaries ultimately used by the server be
returned with this precision (as whether to use local time or UTC hadn't
yet been discussed).  I do think second-level precision could be
reasonable, but if we go this route, I would propose using ISO 8601
timestamps for this purpose, both in the request and the response.  One
happy side-affect here would be that, for most purposes, clients could
still just use years; e.g., "1980" is a valid ISO 8601 date which could be
interpreted by the server as equivalent to "1980-01-01T00:00:00Z".  As long
as the server is clear about what range of valid data is being returned, I
don't see a major problem with this, but we may wish to specify the
intended server behavior here.

We would still need to decide what happens if a specified boundary is
simultaneous with a transition.  I believe the currently described behavior
of inclusive start and exclusive end is alright, but I think this needs to
be made clearer for boundaries with second-level precision.

On 28 August 2014 01:52, Paul Eggert <[email-masked]> wrote:

> However, unlike Ticket 21, I expect the observance bounds should be in
> UTC, not in local time, as this would avoid local-time weirdnesses.

Yes, if we end up agreeing on the first part of this ticket (using UTC for
bounds), I would propose that the observance bounds also be returned in
UTC.  For that matter, if we're heading in this direction, all times,
including transition onsets, should be returned in UTC as ISO 8601
timestamps.  This would balance human- and machine-readability of the JSON
data well, while remaining unambiguous throughout.

--
Tim Parenti

---

From: Paul Eggert <[email-masked]>
Date: Thu, 28 Aug 2014 12:13:59 -0700

Tim Parenti wrote:
> Being more precise about the advertised end date
> only buys upstream data providers an extra 18 days

What?  You want clients to mishandle Marlene Dietrich's time of birth? 
Ohhhh wie schrecklich...!!!  :-)

Admittedly the 18-day gap in 2038 and similar gap in 1901 are not huge 
issues.  However, 32-bit clients should have the same time stamps 
regardless of whether they get their data via tzdist or via traditional 
file-based distribution; anything else is a recipe for confusion.  And 
since this glitch is easily avoidable by fixing the draft protocol, 
let's make sure it is fixed.

---

From: Tim Parenti <[email-masked]>
Date: Thu, 28 Aug 2014 17:42:51 -0400

On 28 August 2014 15:13, Paul Eggert <[email-masked]> wrote:

> However, 32-bit clients should have the same time stamps regardless of
> whether they get their data via tzdist or via traditional file-based
> distribution; anything else is a recipe for confusion.

Yeah, you've convinced me.  I propose we use ISO 8601 for the reasons
mentioned in
http://www.ietf.org/mail-archive/web/tzdist/current/msg00294.html, which
would probably require a new section or subsection detailing how
underspecified timestamps should be expanded.  I haven't read the draft
closely enough to know where that should go, though.

--
Tim Parenti

---

From: Daniel Migault <[email-masked]>
Date: Sat, 30 Aug 2014 21:41:52 +0200

If everyone agree on using ISO 8601. I am closing ticket 21.
BR,
Daniel

On Thu, Aug 28, 2014 at 11:42 PM, Tim Parenti <[email-masked]> wrote:

> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist
>
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Paul Eggert <[email-masked]>
Date: Sat, 30 Aug 2014 13:07:50 -0700

> If everyone agree on using ISO 8601. I am closing ticket 21.

I assume that by ISO 8601 you mean RFC-3339 Section 5.6 date-time? 
That's the ISO 8601 profile for Internet dates.  I suggest adding the 
further constraint that the time-offset must be "Z" or "z".

---

From: Cyrus Daboo <[email-masked]>
Date: Sun, 31 Aug 2014 12:19:49 -0400

Hi Daniel,

> If everyone agree on using ISO 8601. I am closing ticket 21.

Sorry, I have not had a chance to comment on that one yet and do have some 
things to say about it (in particular the usefulness of having the local 
times used in the expand observances results). So can we leave this open 
for a while longer?

-- 
Cyrus Daboo

---

From: Daniel Migault <[email-masked]>
Date: Sun, 31 Aug 2014 20:24:49 +0200

Of course, this issue does not seem completely solved to me. I leave it
open.

On Sun, Aug 31, 2014 at 6:19 PM, Cyrus Daboo <[email-masked]> wrote:

> Hi Daniel,
>
>
> --On August 30, 2014 at 9:41:52 PM +0200 Daniel Migault <
> [email-masked]> wrote:
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58