Subject: [Tzdist] TZSTART/TZEND, was Re: Ticket #21 - proposed changes

From: Cyrus Daboo <[email-masked]>
Date: Fri, 05 Sep 2014 10:56:00 -0400

Hi Lester,

--On September 3, 2014 at 8:20:40 PM +0100 Lester Caine 
<[email-masked]> wrote:

>> I could see either approach being valid, but it seems we should pick
>> one.  I think I know your choice, Lester.  ;)
>
> As I've already said, you need that transition, and you need to flag
> that PRIOR to that transition the tzoffset does not apply, so why not
> just make that the initial truncation? Fixes all the problems?

So on this issue: I had proposed a TZSTART/TZEND to indicate the range of 
validity of an overall VTIMEZONE object. I have thought about this some 
more, in particular with regard to what iCalendar clients currently do, and 
have come to the following conclusions:

1) TZTSTART is not needed. In iCalendar, the STANDARD and DAYLIGHT 
sub-components in a VTIMEZONE are required to have a DTSTART property. That 
means that the lower bound of validity of any VTIMEZONE is simply the 
earliest DTSTART value in any sub-component. Now we could replicate that 
value in the VTIMEZONE itself for clarity, but it would be redundant. That 
does raise the question of what the lower bound should be in some case. 
Here are some possible rules:

    a) For "untruncated" data, the lowest DTSTART value is determined from:
        i) The LMT value specified in the TZDB Zone definition, or
        ii) In the absence of an LMT we pick a value (1883 as the common 
basis for the start of time zones, or 1582 as the basis for the Gregorian 
calendar - which is what iCalendar supports)

    b) For "truncated" data, the lowest DTSTART value is the lower bound 
for truncation. When truncating a time zone we have to insert a "sentinel" 
sub-component at the lower truncation point (assuming it does not match an 
actual valid transition).

2) TZEND is optional and always contains a UTC date-time value. When 
present, it represents the upper bound of validity for the time zone data. 
Clients SHOULD treat any use of this time zone with a date-time value 
(e.g., a DTSTART in a VEVENT) as being invalid. When TZEND is not present, 
that implies that the time zone is unbounded - or the validity beyond the 
last set of transitions, rules is indeterminate. This is basically how 
iCalendar works today, so all we are doing is clarifying the current 
situation.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Fri, 05 Sep 2014 09:07:54 -0700

Cyrus Daboo wrote:

> basis for the start of time zones, or 1582 as the basis for the
> Gregorian calendar - which is what iCalendar supports)

Tim Parenti mentioned this issue too, and I see it's still hanging. 
Here's an example from the tz database:

Zone America/New_York	-4:56:02 -	LMT	1883 Nov 18 12:03:58
			-5:00	US	E%sT	1920
...

This could plausibly be translated to VTIMEZONE in two different ways:

A.  One component with TZOFFSETFROM:-045602, DTSTART:18831118120358, 
TZOFFSETTO:-0500, and TZNAME:EST.

B.  Two components, one the same as (A) and an earlier one with 
TZOFFSETFROM:irrelevant, DTSTART:-infinity, TZOFFSETTO:-045602, and 
TZNAME:LMT.  By "irrelevant" I mean that the number doesn't matter; we 
can suggest "+0000" as a placeholder.  By "-infinity" I mean 
indefinitely into the past; perhaps this should be "00000101" to keep 
iCalendar happy.  (The iCalendar spec doesn't limit dates to 1582 or 
later, as far as I can see; the only limit is the year 0.)

Although (A) is simpler and more compact, (B) contains information that 
(A) lacks, namely the string "LMT".  This string is "zzz" in zones like 
Antarctica/Vostok that were uninhabited before standard time was 
introduced, so there is a smidgen of information there.  So let's 
specify (B).

> for truncation. When truncating a time zone we have to insert a
> "sentinel" sub-component at the lower truncation point (assuming it does
> not match an actual valid transition).

Yes.  This sentinel would have identical TZOFFSETFROM and TZOFFSETTO values.

> present, it represents the upper bound of validity for the time zone
> data. Clients SHOULD treat any use of this time zone with a date-time
> value (e.g., a DTSTART in a VEVENT) as being invalid.

The idea of TZEND makes sense, but I'm afraid I don't understand the 
last sentence quoted above.  I assume it's trying to say that clients 
SHOULD not try to use the time zone past its TZEND boundary?

I see one glitch that is worth documenting.  Suppose the server is 
truncating data for a 32-bit signed POSIX time_t client, so that TZEND 
is 2038-01-19 03:34:08Z.  This time stamp is one second greater than the 
maximum value representable on the client.  The client should be able to 
handle this TZEND value, i.e., it should treat a TZEND value greater 
than what it can represent as if it were infinity, without complaining 
that the TZEND value is too large.

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 05 Sep 2014 12:36:48 -0400

Hi Paul,

--On September 5, 2014 at 9:07:54 AM -0700 Paul Eggert <[email-masked]> 
wrote:

>>     a) For "untruncated" data, the lowest DTSTART value is determined
>>     from: i) The LMT value specified in the TZDB Zone definition, or ii)
>>         In the absence of an LMT we pick a value (1883 as the common
> can suggest "+0000" as a placeholder.  By "-infinity" I mean indefinitely
> into the past; perhaps this should be "00000101" to keep iCalendar happy.
> (The iCalendar spec doesn't limit dates to 1582 or later, as far as I can
> see; the only limit is the year 0.)
>
> introduced, so there is a smidgen of information there.  So let's specify
> (B).

In my opinion the time zone simply did not exist prior to the LMT time 
stamp in the Zone definition, and therefore there should not be any data 
prior to that. As has been pointed out, there is no consistent way to 
indicate what the offset was because it may have varied dramatically from 
(nearby) place to place - and even had multiple values in the same place. 
If we need to handle data prior to the existence of "structured" time 
zones, then I suggest we do that as an extension to the service. e.g., if 
we have a geo: -> time zone data extension, that could be set to return 
multiple items with date ranges applied to each to indicate the range of 
validity of those, with the possibility of overlapping definitions etc. 
Clients would have to choose what they consider appropriate.

> Yes.  This sentinel would have identical TZOFFSETFROM and TZOFFSETTO

Right.

> The idea of TZEND makes sense, but I'm afraid I don't understand the last
> sentence quoted above.  I assume it's trying to say that clients SHOULD
> not try to use the time zone past its TZEND boundary?

Yes. Exactly how the client reacts to that is implementation defined (e.g. 
they could throw up an error, simply not display the event past that time 
etc).

> truncating data for a 32-bit signed POSIX time_t client, so that TZEND is
> 2038-01-19 03:34:08Z.  This time stamp is one second greater than the
> handle this TZEND value, i.e., it should treat a TZEND value greater than
> what it can represent as if it were infinity, without complaining that
> the TZEND value is too large.

In iCalendar, DTEND is treated as an "exclusive" value, whereas DTSTART is 
treated as "inclusive". However, UNTIL in an RRULE is "inclusive" - i.e., 
if the UNTIL value matches an instance generated by the RRULE, that 
instance is treated as valid (basically the last instance in the set). So 
the question is whether we want TZEND to be "inclusive" or "exclusive". If 
the former we might want to call it TZUNTIL instead. In thinking about it 
we might actually want the "inclusive" definition for this. e.g., for a 
zone where the last known transition is:

Rule	Morocco	2037	only	-	Oct	10	 3:00	0	-

The last transition is 20371010T030000 - anything beyond that is not known. 
So what would TZEND be? If it were the "exclusive" value we would have to 
set it at least one second after 20371010T030000. If it were the 
"inclusive" value, then we would set it to the last known transition value. 
>From that perspective I think using the "inclusive" value might make more 
sense, and if we did that, the issue of exceeding the 32-bit time_t range 
would be moot.

-- 
Cyrus Daboo

---

From: Stephen Colebourne <[email-masked]>
Date: Fri, 5 Sep 2014 17:52:10 +0100

On 5 September 2014 17:36, Cyrus Daboo <[email-masked]> wrote:
> In my opinion the time zone simply did not exist prior to the LMT time stamp
> in the Zone definition, and therefore there should not be any data prior to
> that.

Just to note that while this is true in data terms, it isn't what date
and time libraries in most computer languages expect. Most libraries
provide the ability to query an instant with a time-zone into the
far-past. Where they vary is in what (psuedo-meaningless) value they
return. One of the biggest debates in TZDB has been about LMT and more
specifically, the consistency (ie lack of change) of that data.

I've made various suggestions as to mechanisms to provide a
consistent, reliable, never changing value for the far past, but sadly
today the TZDB value is entirely unreliable, changing on a whim. To
emphasise, as a date-time library author, I don't care what the value
is, just that it doesn't change unnecessarily.

Stephen

---

From: Lester Caine <[email-masked]>
Date: Fri, 05 Sep 2014 18:11:11 +0100

> introduced, so there is a smidgen of information there.  So let's

In my way of working, anything prior to the start is LMT, so I'm not
sure we actually need 'B' except to add detail like "zzz". I think that
this area needs to be covered by the location lookup database rather
than tzdist?

My own research is trying to establish a few more date points where an
area started 'standardising' on an average LMT value which does seem to
be the case, so earlier dates may yet be documented. Currently the 'LMT'
point is more likely to only be when they moved from a local time
standard to GMT base standard!

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Paul Eggert <[email-masked]>
Date: Fri, 05 Sep 2014 10:39:45 -0700

Cyrus Daboo wrote:

> the time zone simply did not exist prior to the LMT time
> prior to that.

Unfortunately, as Stephen Colebourne mentioned, date and time libraries 
commonly return *some* value before standard time was introduced; they 
do not simply fail.  The practice has been to return LMT (with an 
approximate UTC offset) for inhabited regions and zzz (with UTC offset 
0) for uninhabited.  tzdist needs to support this practice; otherwise a 
tzdist-supplied client would disagree with a client getting the exact 
same tz database via traditional means, which would be a Bad Thing.

This is related to the problem discussed earlier, with entries like 
Antarctica/Davis that were sporadically uninhabited.  tzdist can simply 
pass through the tz data for that, too; it uses zzz (with UTC offset 0) 
for time stamps while uninhabited.

I expect the tzdist test servers are already behaving this way, and that 
all that's needed is to document it.

> there is no consistent way to
> indicate what the offset was because it may have varied dramatically
> from (nearby) place to place - and even had multiple values in the same
> place.

It's true that this is a problem, but it's not tzdist's problem to 
solve.  tzdist should let a server ship the data it has; it shouldn't 
require the server to truncate its data.

> I think using the "inclusive" value might make more
> sense, and if we did that, the issue of exceeding the 32-bit time_t
> range would be moot.

Thanks, that should work, as long as (and now I'll be even more pickier 
than usual :-) "inclusive" means "up to the end of the named second", 
not "up through the start of the named second".

---

From: Cyrus Daboo <[email-masked]>
Date: Fri, 05 Sep 2014 13:46:19 -0400

Hi Paul,

--On September 5, 2014 at 10:39:45 AM -0700 Paul Eggert 
<[email-masked]> wrote:

> commonly return *some* value before standard time was introduced; they do
> not simply fail.  The practice has been to return LMT (with an
> approximate UTC offset) for inhabited regions and zzz (with UTC offset 0)
> for uninhabited.  tzdist needs to support this practice; otherwise a

Sure. iCalendar itself does not say what clients should do when a time zone 
is used with a date that precedes the earliest defined DTSTART in the 
VTIMEZONE. I suspect many clients will just use the TZOFFSETFROM in the 
earliest observance and extend that back to infinity - I'll have to check 
what my code does.

> than usual :-) "inclusive" means "up to the end of the named second", not
> "up through the start of the named second".

Correct. I'll work on the assumption that we will use TZUNTIL for this 
instead of TZEND.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Fri, 05 Sep 2014 19:51:42 +0100

On 05/09/14 18:46, Cyrus Daboo wrote:
> Sure. iCalendar itself does not say what clients should do when a time
> zone is used with a date that precedes the earliest defined DTSTART in
> the VTIMEZONE. I suspect many clients will just use the TZOFFSETFROM in
> the earliest observance and extend that back to infinity - I'll have to
> check what my code does.

This was one of the areas I was trying to identify. In addition to
identifying periods when the LMT is a local standard, also being able to
identify that a tz IS now inappropriate, and an alternative mechanism
should be used, such as using the location. Prior to probably around
1800 unless you are using a simple longitude LMT for the location, then
it is wrong.

---

From: Tim Parenti <[email-masked]>
Date: Fri, 5 Sep 2014 15:57:31 -0400

On 5 September 2014 13:46, Cyrus Daboo <[email-masked]> wrote:

> zone is used with a date that precedes the earliest defined DTSTART in the

I agree that the cleanest way to represent this, from a tzdist perspective,
is to define the first DTSTART to match the first transition in the source
data (generally a transition away from LMT).  This would include the
TZOFFSETFROM from before that transition, which clients could either extend
back to –infinity, ignore, or replace with some other longitude-based value
they find more suitable for their purpose.  It may be worthwhile to
recommend that clients SHOULD extend this backwards, but ultimately we
can't control what users do with the data — nor should we particularly
care, especially for timestamps that far back.

--
Tim Parenti