Subject: [Tzdist-bis] tzif-04 patches for default time type, etc.

From: Paul Eggert <[email-masked]>
Date: Sun, 3 Jun 2018 20:11:07 -0700

I looked at other implementations and it appears that I was too hasty in my May 
24 comment 
<https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00062.html>. Sorry 
about that. The heuristic that I proposed on May 24 is present in tzcode for 
compatibility with older tzcode versions, and its complexity is not needed for 
currently-generated data and is not used by glibc. This review led me to propose 
today to simplify TZif interpretation slightly in tzdb; see 
<https://mm.icann.org/pipermail/tz/2018-June/026510.html>.

The upshot of this is that for almost all current TZif files, time type 0 should 
be the default time type. The exceptions are limited to a few rarely-used legacy 
zones like EST5EDT that have been superseded by current zones like 
America/New_York, and these few exceptions should vanish in the next tzdb 
release. The attached proposed patch to tzif-04 attempts to take this into 
account, while fixing a few minor omissions that I noticed while I was in the 
neighborhood.

Here are some followup comments and questions.

* Even with the attached patch, the spec requires at least one time type in both 
32- and 64-bit data. However, if there are no transitions and the TZ string is 
nonempty, no time type is used or needed. Should we relax the TZif spec to allow 
no time types in this case? If we do, that would allow slightly-smaller type-4 
TZif files that omit 32-bit data via a dummy with zero time types. A downside, 
though, is that the reference code and glibc currently reject such files, on the 
grounds that tzfile(5) says that there must be at least one time type.

* The spec requires transitions and leap seconds to be listed in "ascending 
order". Does this clearly prohibit duplicate entries? If not, such a prohibition 
should probably be added.

* Should the spec require that a leap second record's 'corr' value differs from 
the preceding record's 'corr' by at most 1? (Or by exactly 1, if no-op records 
are not allowed?) That is how actual leap seconds work, and I suspect (though I 
don't know) that some TZif readers may rely on this.

* Should the spec recommend that every time type is used in at least one 
transition (except that time type 0 need not be used if there is at least one 
transition)? Although any good TZif generator should do this, this is merely an 
optimization (i.e., it omits unnecessary bloat in the TZif file). Similarly for 
the time zone designations array: should the spec recommend that every character 
in that array be used by at least one time type record?
--- draft-murchison-tzdist-tzif-04.xml	2018-06-03 14:55:10.872988614 -0700
+++ draft-murchison-tzdist-tzif-04-fix.xml	2018-06-03 20:04:56.205719211 -0700
@@ -421,6 +421,8 @@
	              type records.
		      The number of type indices is specified by the 'timecnt'
	              field in the header.
+		      Each type index MUST be less than the 'typecnt' field in the
+		      header.
		    </t>
		    <t hangText="local time type records:">
	              A series of six-octet records specifying a local time
@@ -453,11 +455,13 @@
	                  A one-octet unsigned integer specifying an index into
	                  the series of time zone designation characters,
	                  thereby selecting a particular designation string.
+			  This integer MUST be less than the 'charcnt'
+			  field in the header.
	                </t>
	              </list>
	            </t>
		    <t hangText="time zone designations:">
-	              A series of ASCII characters constituting array of
+		      A series of ASCII characters constituting an array of
	              NUL-terminated (0x00) time zone designation strings.
	              The total number of characters is specified by the
	              'charcnt' field in the header.
@@ -539,27 +543,14 @@
	        <t>The type corresponding to a transition time specifies local
	        time for timestamps starting at the given transition time and
	        continuing up to and not including the next transition time.
-	        Local time for timestamps on or after the last transition is
-	        specified by the TZ string in the
-	        <xref target='footer'>footer</xref> if present and nonempty,
-	        and otherwise is specified by the last transition type.
-	        Local time for timestamps before the first transition time (or
-	        for all timestamps if no transitions are present) is specified
-	        by the time type computed from the following rules:
-		<list style='numbers'>
-	          <t>Time type 0 (the first time type) if this time type is
-	          not used in any transitions.</t>
-	
-	          <t>If rule (1) does not apply, time type T if the first
-	          transition is to a daylight saving time type V, T is a
-	          standard time type that is less than V, and no standard time
-	          type U is in the range T &lt; U &lt; V.</t>
-	
-	          <t>If rules (1) and (2) do not apply, the lowest-numbered
-	          standard time type.</t>
-	
-	          <t>If rules (1) through (3) do not apply, time type 0.</t>
-	        </list>
+		Local time on or after the last transition is specified by
+		the footer's TZ string if present and nonempty,
+		and is unspecified otherwise.
+		Time type 0 specifies local time for timestamps before the
+		first transition.
+		If there are no transitions, local time for all timestamps
+		is specified by the TZ string in the footer if present and
+		nonempty, and is time type 0 otherwise.
	        </t>
	
	        <t>

---

From: Ken Murchison <[email-masked]>
Date: Mon, 4 Jun 2018 08:32:08 -0400

On 06/03/2018 11:11 PM, Paul Eggert wrote:
> I looked at other implementations and it appears that I was too hasty 
> in my May 24 comment 
> <https://www.ietf.org/mail-archive/web/tzdist-bis/current/msg00062.html>. 
> Sorry about that. The heuristic that I proposed on May 24 is present 
> in tzcode for compatibility with older tzcode versions, and its 
> complexity is not needed for currently-generated data and is not used 
> by glibc. This review led me to propose today to simplify TZif 
> interpretation slightly in tzdb; see 
> The upshot of this is that for almost all current TZif files, time 
> type 0 should be the default time type. The exceptions are limited to 
> a few rarely-used legacy zones like EST5EDT that have been superseded 
> by current zones like America/New_York, and these few exceptions 
> should vanish in the next tzdb release. The attached proposed patch to 
> tzif-04 attempts to take this into account, while fixing a few minor 
> omissions that I noticed while I was in the neighborhood.

Patch applied with a few edits for consistency.

> * Even with the attached patch, the spec requires at least one time 
> type in both 32- and 64-bit data. However, if there are no transitions 
> and the TZ string is nonempty, no time type is used or needed. Should 
> we relax the TZif spec to allow no time types in this case? If we do, 
> that would allow slightly-smaller type-4 TZif files that omit 32-bit 
> data via a dummy with zero time types. A downside, though, is that the 
> reference code and glibc currently reject such files, on the grounds 
> that tzfile(5) says that there must be at least one time type.

So, would we state that version 1 files MUST include at least one time 
type and that version 2/3 MUST include if the TZ string is not present?  
But, if the latter breaks existing software, perhaps we should leave 
this as-is.

> * The spec requires transitions and leap seconds to be listed in 
> "ascending order". Does this clearly prohibit duplicate entries? If 
> not, such a prohibition should probably be added.

I believe that the definition of ascending would prohibit duplicates, 
but we can add text if we feel that needs to be explicitly spelled out.

> * Should the spec require that a leap second record's 'corr' value 
> differs from the preceding record's 'corr' by at most 1? (Or by 
> exactly 1, if no-op records are not allowed?) That is how actual leap 
> seconds work, and I suspect (though I don't know) that some TZif 
> readers may rely on this.

Do you have suggested text, or should I craft something?

> * Should the spec recommend that every time type is used in at least 
> one transition (except that time type 0 need not be used if there is 
> at least one transition)? Although any good TZif generator should do 
> this, this is merely an optimization (i.e., it omits unnecessary bloat 
> in the TZif file). Similarly for the time zone designations array: 
> should the spec recommend that every character in that array be used 
> by at least one time type record?

We can certainly add text if we deem it necessary, but it seems somewhat 
logical that there is no point in adding unused cruft to the file.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Mon, 11 Jun 2018 12:05:09 -0700

On 06/04/2018 05:32 AM, Ken Murchison wrote:
> type and that version 2/3 MUST include if the TZ string is not 
> present?  But, if the latter breaks existing software, perhaps we 
> should leave this as-is.

OK, but the spec should state this requirement clearly, and mention why 
it's required (as this is not obvious). Proposed patch attached. It 
contains a parenthetical remark that I hope explains things. This patch 
also fixes three other glitches I noticed, as follows.

* TZ strings in footers can be empty, contra POSIX.

* The phrase "time type 0 need not be used if there is at least one 
transition" is incomplete, as time type 0 need not be used even if there 
are no transitions.

* The rule that the TZ string must be consistent with the last 
transition applies only to the last transition in the 64-bit data. The 
TZ string need not be consistent with the last transition in the 32-bit 
data.
--- draft-murchison-tzdist-tzif-05.xml	2018-06-05 17:38:47.247803780 -0700
+++ draft-murchison-tzdist-tzif-05-fix.xml	2018-06-11 11:51:52.097493755 -0700
@@ -314,7 +314,8 @@
			  Version 2 - The file MUST contain both the 32-bit
	                  header and data block and a 64-bit header, data
	                  block, and footer.
-	                  The TZ string in the footer MUST strictly adhere to the
+			  The TZ string in the footer MUST either be empty,
+			  or strictly adhere to the
	                  expanded format of the "TZ" environment variable as
	                  defined in Section 8 of the "Base Definitions"
	                  Volume of <xref target='POSIX'/>.
@@ -323,11 +324,9 @@
			  Version 3 - The file MUST contain both the 32-bit
	                  header and data block and a 64-bit header, data
	                  block, and footer.
-	                  The TZ string in the footer string MAY use <xref
-	                  target='tzstrext'>extensions</xref> to the
-	                  expanded format of the "TZ" environment variable as
-	                  defined in Section 8 of the "Base Definitions"
-	                  Volume of <xref target='POSIX'/>.
+			  The TZ string in the footer MUST strictly adhere
+			  to Version 2 format with the <xref>
+			  target='tzstrext'>extensions described below</xref>.
		        </t>
		      </list>
		    </t>
@@ -353,6 +352,10 @@
	              A four-octet unsigned integer specifying the number of
	              local time type records contained in the data block -
	              MUST NOT be zero.
+		      (Although time type 0 is not used in files that have
+		      nonempty TZ strings but no transitions, it is nevertheless
+		      required because many TZif readers reject files that
+		      lack time types.)
		    </t>
		    <t hangText="charcnt:">
	              A four-octet unsigned integer specifying the total number
@@ -567,9 +570,8 @@
	        "AKST9AKDT").</t>
	
	        <t>In order to eliminate unused space in a TZif file, every
-	        local time type record SHOULD be used by at least one
-	        transition (except that time type 0 need not be used if there
-	        is at least one transition).
+		nonzero local time type index SHOULD appear at least once in the
+		transition type array.
	        Likewise, every character in the time zone designations array
	        SHOULD be used by at least one time type record.</t>
	      </section>  <!-- data -->
@@ -601,8 +603,8 @@
	              of the "Base Definitions" Volume of <xref
	              target="POSIX"/>.  If empty, the corresponding
	              information is not available.
-	              If the string is nonempty, at least one transition MUST
-	              be present in the 64-bit data, and the TZ string MUST be
+		      If the string is nonempty and one or more transitions
+		      appear in the 64-bit data, the string MUST be
	              consistent with the last 64-bit transition - i.e.,
	              evaluating the TZ string at the time of the last
	              transition should yield the same time type as the time