Subject: [Tzdist-bis] tzdist-tzif-01: signed time_t, TZ strings, etc.

From: Paul Eggert <[email-masked]>
Date: Wed, 23 May 2018 12:46:34 -0700

Some random comments on draft 01.

Although POSIX does not specify behavior for negative timestamps, TZif 
files routinely contain them. They represent time before 1970, and the 
spec should reflect common behavior. To fix this, I suggest that we 
change all uses of "POSIX Time" to "Extended POSIX Time", and define the 
latter as "The time as returned by the C time() function (see Section 3 
of the "System Interfaces" Volume of [POSIX]). This is an integer number 
of seconds since the POSIX Epoch (1970-01-01 00:00:00 UTC) not counting 
leap seconds. As an extension to POSIX, negative values represent times 
before the POSIX Epoch, using UTC and its predecessors such as GMT and 
UT." We should also change all existing uses of "epoch" to some other 
word or acronym ("leapt", say) as it's confusing to have "epoch" mean 
two different things. Also, change "292 billion years into the future" 
to "292 billion years into the past or future", and change "no later 
than 19 January, 2038 03:14:07 UTC" to "from 1901-12-13 20:45:52 GMT 
through 2038-01-19 03:14:07 UTC".

The spec doesn't clearly specify that signed values are represented in 
two's complement, or that all bits of the integer representation are 
significant (no padding bits). To fix, how about if we change the note 
about integer representation to: "NOTE: All multi-octet integer values 
MUST be stored in network byte order format (high-order octet first, 
otherwise known as big-endian), with all bits significant. Signed 
integer values MUST be represented using two's complement."

The explanation of standard/wall indicators and UTC/local indicators is 
confusing. It's not clear why these indicators are needed or what they 
do. tzfile(5) says that they are "used when a time zone file is used in 
handling POSIX-style time zone environment variables" and perhaps 
something like this text should be put into the spec.

POSIX does not allow empty TZ strings. The TZif spec does, but is 
incoherent about that. I suggest adding a definition 'TZ String: A 
string that is either empty, or that uses the expanded format of the 
"TZ" environment variable as defined in Section 8 of the "Base 
Definitions" Volume of [POSIX]. If empty, the corresponding information 
is not available.' and then using "TZ string" elsewhere, instead of 
POSIX TZ string, rewording accordingly.

I suggest adding that a TZ string "SHOULD not begin with ':'", since 
POSIX says those strings are implementation-defined.

The spec uses "byte" in a couple of places where I assume "octet" is 
intended.

"times values" should be "time values".

"'leapcnt;" should be "'leapcnt'".

tzfile(5) says "Future changes to the format may append more data." 
Should this intent be specified in TZif?

There should be examples of the two extensions to POSIX in TZ strings. I 
suggest something like the following:

<-03>3<-02>,M3.5.0/-2,M10.5.0/-1

This represents a time zone that observes daylight saving time from 
22:00 on the day before March's last Sunday until 23:00 on the day 
before October's last Sunday. Standard time is 3 hours west of UTC and 
is abbreviated "-03"; daylight saving time is 2 hours west of UTC and is 
abbreviated "-02".

EST5EDT,0/0,J365/25

This represents a time zone that observes daylight saving time all year. 
It is 4 hours west of UTC and is abbreviated "EDT".

---

From: Ken Murchison <[email-masked]>
Date: Wed, 23 May 2018 16:33:33 -0400

Hi Paul,

On 05/23/2018 03:46 PM, Paul Eggert wrote:
> The explanation of standard/wall indicators and UTC/local indicators 
> is confusing. It's not clear why these indicators are needed or what 
> they do. tzfile(5) says that they are "used when a time zone file is 
> used in handling POSIX-style time zone environment variables" and 
> perhaps something like this text should be put into the spec.

I'm not sure what that phrase means.  Can you elaborate?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Wed, 23 May 2018 13:38:51 -0700

> I'm not sure what that phrase means.  Can you elaborate? 

In tzcode-based apps, if you set TZ="AKST9AKDT" (i.e., you say "I'm 
using daylight saving time but I don't want to specify the 
daylight-saving transition rules"), the library implementing localtime 
etc. looks at a TZif file (on my platform, 
/usr/share/zoneinfo/posixrules, which is a copy of 
/usr/share/zoneinfo/America/New_York) to figure out what the transition 
rules ought to be for your incomplete TZ string. To do that, it needs to 
know details about the transitions that ordinarily don't matter, because 
it's inferring the rules from a possibly-unrelated zone (in this case, 
inferring Alaska rules from New York City's). These details are encoded 
in the data at issue. (And I have quite forgotten the details; sorry.)

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2018 09:21:58 -0400

On 23.05.18 15:46, Paul Eggert wrote:
> change all uses of "POSIX Time" to "Extended POSIX Time", and define
> the latter as "The time as returned by the C time() function (see
> Section 3 of the "System Interfaces" Volume of [POSIX]). This is an
> integer number of seconds since the POSIX Epoch (1970-01-01 00:00:00
> UTC) not counting leap seconds. As an extension to POSIX, negative
> values represent times before the POSIX Epoch, using UTC and its
> predecessors such as GMT and UT." We should also change all existing
> uses of "epoch" to some other word or acronym ("leapt", say) as it's
> confusing to have "epoch" mean two different things. Also, change "292
> billion years into the future" to "292 billion years into the past or
> future", and change "no later than 19 January, 2038 03:14:07 UTC" to
> "from 1901-12-13 20:45:52 GMT through 2038-01-19 03:14:07 UTC".

I think this is fine, but we might need to liaise to P.1003/OpenGroup to
let them know that we're doing it (Alexey/mnot?).  We may also want to
call it something other than "Extended POSIX" but rather something like
"TZ time format", so as to avoid confusion.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 May 2018 09:46:11 -0400

> "TZ time format", so as to avoid confusion.

Can we use something like "UNIX time"?  There is a Wikipedia entry for 
it which discusses negative values.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 May 2018 09:58:03 -0400

> Should this intent be specified in TZif?

I don't think our document prevents appending or removing data, 
correct?  If we think that we need to explicitly state that this is 
allowed, I can add some text.

Actually, while we're at it, do we want to define a version 4, which 
drops the 32-bit header and data block?  Of course, doing this would 
require another reorganization of the document.  Perhaps, this isn't the 
right time for this.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2018 10:03:07 -0400

> it which discusses negative values.

We need either a solid normative reference or a solid definition.  For a
normative reference it has to be stable.  Wikipedia cannot be considered
stable.  IEEE specs are.

Eliot

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2018 10:05:21 -0400

On 24.05.18 09:58, Ken Murchison wrote:
> require another reorganization of the document.  Perhaps, this isn't
> the right time for this.
>

I was wondering that myself, but my issue is that TZ has a great track
record of having running code for such things, and such a version
wouldn't have running code.  Still it's worth thinking about.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 May 2018 10:05:25 -0400

> normative reference it has to be stable.  Wikipedia cannot be considered

I wasn't anticipating using Wikipedia as the reference, just using a 
fairly well-known term such as "UNIX Time" rather than having to create 
our own.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2018 10:32:49 -0400

Ok that's fine.

On 24.05.18 10:05, Ken Murchison wrote:
> fairly well-known term such as "UNIX Time" rather than having to
> create our own.
>

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 May 2018 11:57:20 -0400

I believe that version -02 addresses all of these issues, other than 
whether we to need to explicitly state that future versions of TZif may 
add additional data.

On 05/23/2018 03:46 PM, Paul Eggert wrote:
> Definitions" Volume of [POSIX]. If empty, the corresponding 
> information is not available.' and then using "TZ string" elsewhere, 
> instead of POSIX TZ string, rewording accordingly.
>
> There should be examples of the two extensions to POSIX in TZ strings. 
> I suggest something like the following:
>
> is abbreviated "-03"; daylight saving time is 2 hours west of UTC and 
> is abbreviated "-02".
>
> EST5EDT,0/0,J365/25
>
> This represents a time zone that observes daylight saving time all 
> year. It is 4 hours west of UTC and is abbreviated "EDT".
>
> _______________________________________________
> Tzdist-bis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist-bis

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Thu, 24 May 2018 17:04:10 -0700

On 05/24/2018 07:05 AM, Eliot Lear wrote:
> I was wondering that myself, but my issue is that TZ has a great track  > record of having running code for such things, and such a version > 
wouldn't have running code. Still it's worth thinking about.
I thought about it a bit. Presumably version-4 format files would would 
still work with existing clients that grok only earlier versions.

For clients that support version 2 or 3, a version 4 file would look 
like the 32-bit data were size zero. A 64-bit client should work just 
fine with version 4, because the 64-bit data is unaffected. A 32-bit 
client should either (a) fall back on the TZ string (good enough for 
recent and future timestamps), or (b) scan the 64-bit data and discard 
data that doesn't fit into 32 bits. Although the most recent release of 
the GNU C Library does (a), the next release is planed to switch to (b) 
as per this patch dated 2018-05-11:

https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fc79706a3233b02909b1a75b1e43e40915eb5a04

It does strike me that the TZif spec needs some wording to the effect 
that the 32-bit data should be a consistent subset of the 64-bit data. 
Right now, the spec allows them to differ completely from each other, 
but that's not the intent and it's not common practice. OTOH we do want 
to allow the subset to be drastically pruned down in version 4.....

All in all I'm inclined to take this matter up with the tz mailing list, 
and to defer version 4 to a later edition of the TZif spec. In the mean 
time we need to think about adding the wording along the lines mentioned 
above.

---

From: Ken Murchison <[email-masked]>
Date: Thu, 24 May 2018 20:23:04 -0400

On 05/24/2018 08:04 PM, Paul Eggert wrote:
>> I was wondering that myself, but my issue is that TZ has a great 
>> track  > record of having running code for such things, and such a 
>> version > 
> I thought about it a bit. Presumably version-4 format files would 
> would still work with existing clients that grok only earlier versions.
>
> data that doesn't fit into 32 bits. Although the most recent release 
> of the GNU C Library does (a), the next release is planed to switch to 
> (b) as per this patch dated 2018-05-11:
>
> but that's not the intent and it's not common practice. OTOH we do 
> want to allow the subset to be drastically pruned down in version 4.....
>
> All in all I'm inclined to take this matter up with the tz mailing 
> list, and to defer version 4 to a later edition of the TZif spec. In 
> the mean time we need to think about adding the wording along the 
> lines mentioned above.

We already have this courtesy of Eliot's previous siggestion:

"Implementations SHOULD generate version 2 or 3 files, and SHOULD strive 
to make the pre-2038 transition times in the 32-bit and 64-bit data 
blocks consistent with one another."

Do you want something stronger?  Do your have suggested text?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Fri, 25 May 2018 15:09:15 -0700

On 05/24/2018 05:23 PM, Ken Murchison wrote:
> "Implementations SHOULD generate version 2 or 3 files, and SHOULD 
> strive to make the pre-2038 transition times in the 32-bit and 64-bit 
> data blocks consistent with one another."
>
>

Thanks, I hadn't noticed that change. It's a start, but more work is 
needed. For example:

* It's not just transition times; it's all the contents of the data 
block and of the corresponding header.

* If we do version 4 along the lines suggested, version 4 files will 
have 32-bit sections that are essentially empty. Would that be 
"consistent" with the 64-bit data (because it would contain no 
transitions that disagree with the 64-bit data), or would it be 
"inconsistent" (because it would lack 64-bit transitions that fall into 
the 32-bit space)?

* As another example, suppose the 32-bit data contain only transitions 
during years 1970-2038 and the 64-data contain all known transitions 
including those in the 1901-1969 year range. (zic has never done this, 
but suppose it optionally did so, to try to cater to buggy old 32-bit 
clients.) Would that be "consistent", even though negative timestamps in 
the 32-bit range would be treated differently depending on whether the 
implementation used the 32-bit or the 64-bit data?

With the above in mind, how about if we replace the abovequoted text 
with the following: "Implementations SHOULD generate version 2 or 3 
files. The 64-bit header and data block SHOULD contain a superset of the 
time zone information in the 32-bit header and data block. When reading 
a version 2 or 3 file, implementations SHOULD ignore the 32-bit header 
and data block except for the purpose of skipping over them."

---

From: Eliot Lear <[email-masked]>
Date: Fri, 25 May 2018 19:43:40 -0400

Hi Paul,.

On 25.05.18 18:09, Paul Eggert wrote:
> have 32-bit sections that are essentially empty. 

Why not indicate '4' in the first version field and let that be the only
header?  E.g., just proceed with 64 bit data?

> Would that be "consistent" with the 64-bit data (because it would
> contain no transitions that disagree with the 64-bit data), or would
> it be "inconsistent" (because it would lack 64-bit transitions that
> fall into the 32-bit space)?
>
> clients.) Would that be "consistent", even though negative timestamps
> in the 32-bit range would be treated differently depending on whether
> the implementation used the 32-bit or the 64-bit data?
>
> files. 
I would put this in the negative to future proof: Implementations SHOULD
NOT generate version 1 files.

> The 64-bit header and data block SHOULD contain a superset of the time
> zone information in the 32-bit header and data block.

Then we would have to explain what a superset means.  You could say:
transitions that are present in the 32-bit section MUST be semantically
identical in the 64-bit section.  The 64-bit section MAY include
transitions that precede January 1, 1970 and are later than January
18th, 2038.

> When reading a version 2 or 3 file, implementations SHOULD ignore the
> 32-bit header and data block except for the purpose of skipping over
> them."

s/SHOULD/MUST/?

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Fri, 25 May 2018 18:19:25 -0700

> header?  E.g., just proceed with 64 bit data?

Because that would likely break version 1 or 2 or 3 apps that attempt to 
read version 4 data. So far, TZif formats have always had the property 
that later-version TZif files are typically useful even to 
earlier-version TZif readers, which has made upgrading easier. If it's 
easy to keep this property for version 4, which I suspect it is, I'd 
rather keep it.

> NOT generate version 1 files.

My understanding (though I can by no means claim to know all 
implementations) is that the main remaining system using 32-bit-only 
data is Android. Android concatenates a bunch of 32-bit TZif-format 
files together into a single file with a small header and an associated 
index file, and uses this regardless of whether the application is 32- 
or 64-bit. For an example of the single file, see 
<https://android.googlesource.com/platform/system/timezone/+/a591fb03ae2beb5c5bc8696eaf64e67cc0c73fc1/output_data/iana/tzdata>. 
Should we be concerned about this common existing practice enough to 
document it too?

Neil Fuller of Google is probably the best guy to talk to about this. 
Perhaps we should be attempting to bring him into the conversation, 
because Google might be interested in using tzdist to transmit its 
tzdata files.

> transitions that precede January 1, 1970 and are later than January

Yes, we could work on text along those lines, though then the question 
would then devolve to "what do we mean by 'semantically identical'?".  
For example, does a no-op transition (one where the UT offset and 
abbreviation and daylight-saving flags do not change) contribute to 
semantics? Also, my thought is that in version 4 the 64-bit section will 
typically include transitions in the 1970-2038 range that are not in the 
(empty) 32-bit section, so the above wording is too restrictive.

>> 32-bit header and data block except for the purpose of skipping over

In some current implementations, 32-bit data override 64-bit data on 
32-bit clients. As I mentioned the GNU C Library is one example (though 
this has been patched in master and should be fixed in a future 
release). So I'm a bit reluctant to use MUST here.

---

From: Ken Murchison <[email-masked]>
Date: Sat, 26 May 2018 07:44:10 -0400

On 05/25/2018 09:19 PM, Paul Eggert wrote:
> Because that would likely break version 1 or 2 or 3 apps that attempt 
> to read version 4 data. So far, TZif formats have always had the 
> property that later-version TZif files are typically useful even to 

We already have this sentence:

       "Version 1 files are considered a legacy format and SHOULD NOT
       be generated, as they do not support transition times after the
       year 2038."

> files together into a single file with a small header and an 
> associated index file, and uses this regardless of whether the 
> application is 32- or 64-bit. For an example of the single file, see 

This almost seems like a different format and if its unique to Google, 
perhaps they should document it.  Do we know of any other software that 
reads this blob?  Also, tzdist currently distributes data on a 
per-timezone basis, NOT the complete database as a blob.

> semantics? Also, my thought is that in version 4 the 64-bit section 
> will typically include transitions in the 1970-2038 range that are not 
> in the (empty) 32-bit section, so the above wording is too restrictive.
>
> 32-bit clients. As I mentioned the GNU C Library is one example 
> (though this has been patched in master and should be fixed in a 
> future release). So I'm a bit reluctant to use MUST here.

I'm starting to wonder if even recommending consistency between 32-bit 
data and 64-bit data is something we want to get into at this point.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Sat, 26 May 2018 09:05:19 -0400

Hi!

On 26.05.18 07:44, Ken Murchison wrote:
>> earlier-version TZif readers, which has made upgrading easier. If
>> it's easy to keep this property for version 4, which I suspect it is,
>> I'd rather keep it.
>>

Sorry- missed that.

> perhaps they should document it.  Do we know of any other software
> that reads this blob?  Also, tzdist currently distributes data on a

Right.

> [...]

I'm ok with keeping it a SHOULD but it's important to provide guidance
here.  Otherwise we'll end up with inconsistent handling.

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Sat, 26 May 2018 11:25:18 -0700

> here.  Otherwise we'll end up with inconsistent handling.

Yes. In practice the intent has always been that the data are consistent, and 
clients in practice rely on this intent. The devil is in the details, as there 
have been be minor differences (e.g., no-op transitions, or data just before the 
2038 cutoff that is purposely omitted in the 32-bit data to avoid confusion) 
that are OK in the context of the overall intent. The wording I proposed at the 
end of <https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00032.html> 
attempted to thread the needle of recording this intent as a SHOULD without 
going into too much technical detail. But if we'd like more detail, how about 
the something like the following wording instead:

Replace:

"Implementations SHOULD generate version 2 or 3 files, and SHOULD strive to make 
the pre-2038 transition times in the 32-bit and 64-bit data blocks consistent 
with one another."

with

"Implementations SHOULD generate version 2 or 3 files. The sequence of time 
changes defined by the 32-bit header and data block SHOULD be a contiguous 
subsequence of the time changes defined by the 64-bit header and data block. 
When reading a version 2 or 3 file, implementations SHOULD ignore the 32-bit 
header and data block except for the purpose of skipping over them."

and add the following to the definitions:

"Time Change: A change to civil timekeeping practice. It occurs when one or more 
of the following happen simultaneously: (1) a change in UTC offset, (2) a change 
in whether standard or daylight saving time is in use, (3) a change in time zone 
abbreviation, and (4) a leap second (i.e., a change in UTC - TAI)."

and insert the word "may" just before the last word ("change") in the 
"transition times:" sentence "Each value is used as a transition time at which 
the rules for computing local time change."

---

From: Eliot Lear <[email-masked]>
Date: Mon, 28 May 2018 08:15:43 +0200

Just for planning purposes, backing up to the question of how to
handle/plan for v4:

> have 32-bit sections that are essentially empty.

If we do that, the following text in -02 has to be reviewed:

   typecnt:  A four-octet unsigned integer specifying the number of
      local time type Records contained in the data block - MUST NOT be
      zero.

Do we want to say that it "MUST NOT be zero for versions described in
this document"?

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Sun, 27 May 2018 23:43:50 -0700

>        local time type Records contained in the data block - MUST NOT be

I'm not offhand familiar with this constraint. Why does it say "MUST 
NOT" be zero now?  What would break if it's zero, and why?

---

From: Lester Caine <[email-masked]>
Date: Mon, 28 May 2018 09:56:55 +0100

> Just for planning purposes, backing up to the question of how to

A more fundamental question is if the blind move to simply making 32bit 
numbers into 64bit is the right answer at all? While a lot of computer 
hardware will be handling 64bit numbers natively, the vast majority of 
kit that is part of the 'IOT' drive will probably be 32bit or less, so 
handling this will need additional programming. A switch to a V4 that is 
still 32bit, but allows for a bit more flexibility should at least be 
considered? DROP the reliance on second based calendars and switch to 
day based allows a large number of simple operations to be tidied up. 
Make 'day' and 'time' two separate 32bit fields and allow the two to be 
managed independently. This sidesteps the problem of leap seconds, and 
allows different calendars to display date without having to worry about 
them. The time element is still managed as a seconds value, but can be 
treated as a fraction of a day as well with only leap-second days 
needing any special processing. This also opens the door for IOT 16bit 
devices to simply trim their date range and work with minute accuracy 
for time as many devices currently do. It is these sort of devices that 
will benefit most from a binary tzdist feed of a single timezone 
rule-set, or even just an appropriately truncated rule-set.

To that end, I don't see why the leapsecond dataset should be included 
in every download. It should be just an optional download.

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Eliot Lear <[email-masked]>
Date: Mon, 28 May 2018 12:21:22 +0200

Lester,

I think you just made a great argument for NOT including v4 in this
round.  It could be that we do both a v4 and v5, even.  I *could*
envision doing something where we dinked with the value of Epoch, but
that would be complex processing at the expense of 64-bit formats.

My point was simple: if we would have any format that requires a header
and then an empty block, we can't have an empty block if one of the
count values is > 0.

Eliot

On 28.05.18 10:56, Lester Caine wrote:
> A more fundamental question is if the blind move to simply making
> 32bit numbers into 64bit is the right answer at all? While a lot of
> computer hardware will be handling 64bit numbers natively, the vast
> majority of kit that is part of the 'IOT' drive will probably be 32bit
> or less, so handling this will need additional programming. A switch
> to a V4 that is still 32bit, but allows for a bit more flexibility
> should at least be considered? DROP the reliance on second based
> calendars and switch to day based allows a large number of simple
> operations to be tidied up. Make 'day' and 'time' two separate 32bit
> fields and allow the two to be managed independently. This sidesteps
> the problem of leap seconds, and allows different calendars to display
> date without having to worry about them. The time element is still
> managed as a seconds value, but can be treated as a fraction of a day
> as well with only leap-second days needing any special processing.
> This also opens the door for IOT 16bit devices to simply trim their
> date range and work with minute accuracy for time as many devices
> currently do. It is these sort of devices that will benefit most from
> a binary tzdist feed of a single timezone rule-set, or even just an
> appropriately truncated rule-set.
>

---

From: Lester Caine <[email-masked]>
Date: Mon, 28 May 2018 12:27:00 +0100

> that would be complex processing at the expense of 64-bit formats.
It is still another 20 years until 2038 capability becomes essential, 
and I hope to still be around then, but modifying a single data block to 
expand the range does not need to be at the 'expense' of 64bit data. One 
simply combines two 32 bit fields to provide the current 64 bit value? 
And the client end decides if that needs to be modified by leapsecond or 
not! Making everything ONLY 64bit is at the expense of every other 
non-64bit kit is the problem here? While providing a single binary 
rule-set that anything can use would seem a much more sensible way forward.

> and then an empty block, we can't have an empty block if one of the
Given that many time zone rule-sets are a single fixed offset, a number 
of the proposed data blocks are empty? The example target of an IOT 
device may only require current local time and the time of the next 
offset change which may well be zero. Although a leap second offset 
could possibly be handled that way?

I am still struggling to see where tzif actually fits in with tzdist. 
It's not entirely clear that the current draft IS only for a single 
rule-set? I seem to remember the discussion on tzdist was for a binary 
dump of the entire database ... downloading leapseconds with every 
rule-set just seems overkill in that case.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 28 May 2018 19:28:27 +0200

On 28.05.18 13:27, Lester Caine wrote:
> and I hope to still be around then, but modifying a single data block
> to expand the range does not need to be at the 'expense' of 64bit data.

No you misunderstand my point: if we're really worried about processing
64 bit data, then we have to add some complexity in terms of
interpreting 32 bit data, or we simply leave the various data sets in
there, in which case we worry about space.

> One simply combines two 32 bit fields to provide the current 64 bit
> value? 

Sure.  That's one way to go.

> And the client end decides if that needs to be modified by leapsecond
> or not! Making everything ONLY 64bit is at the expense of every other

Sure, if we can agree which one.  I'm just saying that we don't need to
do so now, especially if it's going to hold things up.  And personally I
don't think 64-bit processing in these cases is all that difficult,
seeing as long long has been around since 1989.

> Given that many time zone rule-sets are a single fixed offset, a
> number of the proposed data blocks are empty? The example target of an
> IOT device may only require current local time and the time of the
> next offset change which may well be zero. Although a leap second
> offset could possibly be handled that way?

There are a good number of ways one can optimize those devices.  Here
are three right off the top of my head:

  * Only store localtime and the next transition and offset.
  * Only store the TZ string and process that.
  * Retain only transition information back as far as the device has
    operated and discard the rest.

All of that will depend on the way the device is intended to be used.  I
don't think we have to worry about any of that.  Which brings us to your

> rule-set just seems overkill in that case.

If it's not clear we should fix that.  In fact I do not see how one
could use this format to download every ruleset in one go.  But I also
think you've hit upon another point: the draft does not define a term
for what this binary blob is.  7808 only defines the components.

Eliot

---

From: Michael Douglass <[email-masked]>
Date: Mon, 28 May 2018 13:53:10 -0400

On 5/28/18 04:56, Lester Caine wrote:
> or less, so handling this will need additional programming. 
This raises a very good point. We need to do more to address the needs 
of small devices. That's actually more or less where we started when the 
early mobile phones only had space for a few timezones. That issue went 
away but the IOT one probably won't.

Accuracy to the minute is plenty for many devices and supporting such a 
format is probably more pressing than 64 bit

> A switch to a V4 that is still 32bit, but allows for a bit more 
> flexibility should at least be considered? DROP the reliance on second 
> based calendars and switch to day based allows a large number of 
> simple operations to be tidied up. Make 'day' and 'time' two separate 
> 32bit fields and allow the two to be managed independently. This 
> sidesteps the problem of leap seconds, and allows different calendars 
> to display date without having to worry about them. The time element 
> is still managed as a seconds value, but can be treated as a fraction 
> of a day as well with only leap-second days needing any special 
> processing. This also opens the door for IOT 16bit devices to simply 
> trim their date range and work with minute accuracy for time as many 
> devices currently do. It is these sort of devices that will benefit 
> most from a binary tzdist feed of a single timezone rule-set, or even 
> just an appropriately truncated rule-set.
>

---

From: Paul Eggert <[email-masked]>
Date: Mon, 28 May 2018 12:38:00 -0700

On 05/28/2018 10:28 AM, Eliot Lear wrote:

> I don't think 64-bit processing in these cases is all that difficult, 

That's right. The trend in recent 32-bit platforms is to use 64-bit 
signed time_t. 32-bit Microsoft Windows does this, as does 32-bit 
NetBSD, OpenBSD, FreeBSD (except i386), GNU/Linux x32, and others. We 
are certainly not there universally (e.g., GNU/Linux i686 is still 
32-bit time_t, and patches to fix this in glibc are in the works but 
have not been installed) but we will have to be there before 2038 rolls 
around, as the other plausible alternatives (TZif files that assume 
unsigned time_t, or that change the Epoch) are not used in practice and 
are not likely to be used in the future largely due to compatibility 
hassles.

"32-bit" typically refers to the width of pointers, not integers, and it 
isn't much trouble nowadays to support 64-bit time_t even on 32-bit 
platforms.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 28 May 2018 12:48:09 -0700

On 05/28/2018 10:28 AM, Eliot Lear wrote:
> the draft does not define a term for what this binary blob is

In thinking about it a bit more, the "binary blob" is better thought of 
as the zic source file, which is called tzdata.zi in the current 
distribution. This file contains all the information in the entire 
database. It is currently about 104 KiB in text form and about 21 KiB 
compressed, so it's reasonable to retransmit the whole thing whenever 
any zone changes.

Presumably this file would be a different file type, not TZif, and 
specifying it would be the job of a different draft, not the TZif draft.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 28 May 2018 13:12:26 -0700

> NOT" be zero now?  What would break if it's zero, and why? 

I looked into this some more, and the "MUST NOT" has been documented in 
tzfile.5 for three decades. It was put in there because of the issue of 
time stamps before the first transition, a  topic that the TZif spec 
needs to address. I plan to start a new thread about this.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 29 May 2018 08:28:01 -0400

> NOT" be zero now?  What would break if it's zero, and why?

That restraint came  from tzfile(5).  I have no idea why it is there.  
I'm happy to remove it.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Tue, 29 May 2018 11:18:00 -0400

Done.

On 05/26/2018 02:25 PM, Paul Eggert wrote:
> Yes. In practice the intent has always been that the data are 
> consistent, and clients in practice rely on this intent. The devil is 
> in the details, as there have been be minor differences (e.g., no-op 
> transitions, or data just before the 2038 cutoff that is purposely 
> omitted in the 32-bit data to avoid confusion) that are OK in the 
> context of the overall intent. The wording I proposed at the end of 
> <https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00032.html> 
> attempted to thread the needle of recording this intent as a SHOULD 
> without going into too much technical detail. But if we'd like more 
> detail, how about the something like the following wording instead:
>
> Replace:
>
> "Implementations SHOULD generate version 2 or 3 files. The sequence of 
> time changes defined by the 32-bit header and data block SHOULD be a 
> contiguous subsequence of the time changes defined by the 64-bit 
> header and data block. When reading a version 2 or 3 file, 
> implementations SHOULD ignore the 32-bit header and data block except 
> for the purpose of skipping over them."
>
> "Time Change: A change to civil timekeeping practice. It occurs when 
> one or more of the following happen simultaneously: (1) a change in 
> UTC offset, (2) a change in whether standard or daylight saving time 
> is in use, (3) a change in time zone abbreviation, and (4) a leap 
> second (i.e., a change in UTC - TAI)."
>
> "transition times:" sentence "Each value is used as a transition time 
> at which the rules for computing local time change."

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC