Subject: [Tzdist-bis] tzif-05 patches for sanity checks and correct TAI

From: Paul Eggert <[email-masked]>
Date: Mon, 11 Jun 2018 13:52:08 -0700

I looked through the reference and glibc implementations for sanity 
checks, which cause these libraries to reject TZif files as invalid, and 
came up with the attached patches which try to record each sanity check 
one or the other implementations make. Although there is no explicit 
sanity check for -2**31, code seems to negate that value with abandon 
(and this has undefined behavior in C) so I felt it wise to require that 
it be negatable. The 26-hour limit comes from POSIX's maximum offset of 
25:59:59.

In the process of doing this I discovered that the spec's definition of 
"leap second correction" is vague and arguably wrong. The value recorded 
a TZif file is TAI - UTC - 10, but the spec talks only about UTC - TAI. 
The attached patch also tries to clear this matter up. It's a bit 
complicated, since TAI is not well-defined for older timestamps (and the 
exact definition of "old" here is a bit blurry, but "before 1972" is 
good enough for this spec).

--- draft-murchison-tzdist-tzif-05-fix.xml	2018-06-11 13:10:03.944359456 -0700
+++ draft-murchison-tzdist-tzif-05-fix2.xml	2018-06-11 13:43:57.739617385 -0700
@@ -149,7 +149,7 @@
	          may be positive, negative, or zero.
	        </t>
	        <t hangText="International Atomic Time (TAI):">
-	          The time standard based on atomic clocks.
+		  The time standard based on atomic clocks since 1972.
	          It is equal to UTC except without leap second adjustments.
	        </t>
	        <t hangText="Local Time:">
@@ -445,6 +445,9 @@
	                  A four-octet signed integer specifying the number of
	                  seconds to be added to UT in order to determine local
	                  time.
+			  The value MUST not be -2**31, and SHOULD be in
+			  the range [-93599, 93599] (i.e., its absolute
+			  value SHOULD be less than 26 hours).
	                </t>
		        <t hangText="(is)dst:">
	                  A one-octet value indicating whether local
@@ -452,6 +455,7 @@
	                  A value of one (1) indicates that DST is in effect.
	                  A value of zero (0) indicates that standard time in
	                  effect.
+			  The value MUST be 0 or 1.
	                </t>
		        <t hangText="(desig)idx:">
	                  A one-octet unsigned integer specifying an index into
@@ -477,6 +481,11 @@
	              strictly ascending order.
	              The number of records is specified by the
	              'leapcnt' field in the header.
+		      If no records are present, or if a timestamp occurs
+		      before the occurrence time in the first record, the
+		      value of TAI - UTC - 10 is zero, and this zero value
+		      applies proleptically even to timestamps before the
+		      introduction of TAI or UTC.
	              Each record has one of the following structures:
		      <list style='hanging'>
		        <t hangText="32-bit Data Block:">
@@ -502,11 +511,17 @@
		        <t hangText="occur(rence):">
	                  A four- or eight-octet UNIX time value specifying the
	                  time at which a leap second correction occurs.
+			  The first value, if present, MUST be nonnegative,
+			  and each later value MUST be at least 2419199
+			  greater than the previous value.
+			  (This is 28 days' worth of seconds, minus a
+			  potential negative leap second.)
	                </t>
		        <t hangText="corr(ection):">
-	                  A four-octet signed integer specifying the total
-	                  number of leap seconds to be applied to UTC on or
-	                  after the occurrence.
+	                  A four-octet signed integer specifying the value
+			  of TAI - UTC - 10 on or after the occurrence.
+			  The correction value in the first leap second record,
+			  if present, MUST be either one (1) or minus one (-1).
	                  The correction values in adjacent leap second
	                  records MUST differ by exactly one (1).
	                </t>
@@ -516,6 +531,7 @@
	              A series of one-octet values indicating whether
	              the transition times associated with local time types were
	              specified as standard time or wall clock time.
+		      Each value MUST be 0 or 1.
	              A value of one (1) indicates standard time,
	              and MUST be set to one (1) if the corresponding
	              UT/local indicator is set to one (1).
@@ -530,6 +546,7 @@
	              A series of one-octet values indicating whether
	              the transition times associated with local time types were
	              specified as UT or local time.
+		      Each value MUST be 0 or 1.
	              A value of one (1) indicates UT,
	              and the corresponding standard/wall indicator MUST also
	              be set to one (1).

---

From: Eliot Lear <[email-masked]>
Date: Tue, 12 Jun 2018 07:35:36 -0400

Paul,

Let me suggest that the explanatory text needs to be in the draft as
well.  The patch, as it stands, is opaque.

Eliot

On 11.06.18 16:52, Paul Eggert wrote:
> checks, which cause these libraries to reject TZif files as invalid,
> and came up with the attached patches which try to record each sanity
> check one or the other implementations make. Although there is no
> explicit sanity check for -2**31, code seems to negate that value with
> abandon (and this has undefined behavior in C) so I felt it wise to
> require that it be negatable. The 26-hour limit comes from POSIX's
> maximum offset of 25:59:59.
>
> In the process of doing this I discovered that the spec's definition
> of "leap second correction" is vague and arguably wrong. The value
> recorded a TZif file is TAI - UTC - 10, but the spec talks only about
> UTC - TAI. The attached patch also tries to clear this matter up. It's
> a bit complicated, since TAI is not well-defined for older timestamps
> (and the exact definition of "old" here is a bit blurry, but "before
> 1972" is good enough for this spec).
>
>
>
> _______________________________________________
> Tzdist-bis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist-bis

---

From: Paul Eggert <[email-masked]>
Date: Tue, 12 Jun 2018 18:56:51 -0700

> well.  The patch, as it stands, is opaque.

Yes, it could use some more explanation. (For one thing, the word 
"proleptic" stuck out like a sore thumb....) Revised proposed patch 
attached, which I hope is less opaque.

This patch tries to address a further issue that came up in the tz 
mailing list yesterday, by adding a suggestion that transition times 
SHOULD not be before the Big Bang (a rule that's been in zic.c for a 
couple of years, I think, to work around bugs in glib and in glibc). 
Also, it tightens the suggested UT offset range from [-93599, 93599] to 
[-89999, 93599] since that's what POSIX requires.

--- draft-murchison-tzdist-tzif-05.xml	2018-06-11 13:00:12.145631110 -0700
+++ draft-murchison-tzdist-tzif-05-fix3.xml	2018-06-12 18:47:05.219219428 -0700
@@ -149,7 +149,7 @@
	          may be positive, negative, or zero.
	        </t>
	        <t hangText="International Atomic Time (TAI):">
-	          The time standard based on atomic clocks.
+		  The time standard based on atomic clocks since 1972.
	          It is equal to UTC except without leap second adjustments.
	        </t>
	        <t hangText="Local Time:">
@@ -314,7 +314,8 @@
			  Version 2 - The file MUST contain both the 32-bit
	                  header and data block and a 64-bit header, data
	                  block, and footer.
-	                  The TZ string in the footer MUST strictly adhere to the
+			  The TZ string in the footer MUST either be empty,
+			  or strictly adhere to the
	                  expanded format of the "TZ" environment variable as
	                  defined in Section 8 of the "Base Definitions"
	                  Volume of <xref target='POSIX'/>.
@@ -323,11 +324,9 @@
			  Version 3 - The file MUST contain both the 32-bit
	                  header and data block and a 64-bit header, data
	                  block, and footer.
-	                  The TZ string in the footer string MAY use <xref
-	                  target='tzstrext'>extensions</xref> to the
-	                  expanded format of the "TZ" environment variable as
-	                  defined in Section 8 of the "Base Definitions"
-	                  Volume of <xref target='POSIX'/>.
+			  The TZ string in the footer MUST strictly adhere
+			  to Version 2 format with the <xref>
+			  target='tzstrext'>extensions described below</xref>.
		        </t>
		      </list>
		    </t>
@@ -353,6 +352,10 @@
	              A four-octet unsigned integer specifying the number of
	              local time type records contained in the data block -
	              MUST NOT be zero.
+		      (Although time type 0 is not used in files that have
+		      nonempty TZ strings but no transitions, it is nevertheless
+		      required because many TZif readers reject files that
+		      lack time types.)
		    </t>
		    <t hangText="charcnt:">
	              A four-octet unsigned integer specifying the total number
@@ -413,6 +416,11 @@
	              for computing local time may change.
		      The number of time values is specified by the 'timecnt'
	              field in the header.
+		      Each time value SHOULD be at least -2**59.
+		      (-2**59 is the greatest negated power of 2 that predates
+		      the Big Bang, and avoiding earlier timestamps works
+		      around known TZif reader bugs relating to outlandlishly
+		      negative timestamps.)
		    </t>
		    <t hangText="transition types:">
		      A series of one-octet unsigned integers specifying the
@@ -442,6 +450,15 @@
	                  A four-octet signed integer specifying the number of
	                  seconds to be added to UT in order to determine local
	                  time.
+			  The value MUST NOT be -2**31, and SHOULD be in
+			  the range [-89999, 93599] (i.e., its absolute
+			  value SHOULD be more than -25 hours and less than
+			  26 hours).
+			  (Avoiding -2**31 allows 32-bit clients to negate
+			  the value without overflow.
+			  Restricting it to [-93599, 93599] allows easy support by
+			  implementations that already support the
+			  the POSIX-required range [-24:59:59, 25:59:59].)
	                </t>
		        <t hangText="(is)dst:">
	                  A one-octet value indicating whether local
@@ -449,6 +466,7 @@
	                  A value of one (1) indicates that DST is in effect.
	                  A value of zero (0) indicates that standard time in
	                  effect.
+			  The value MUST be 0 or 1.
	                </t>
		        <t hangText="(desig)idx:">
	                  A one-octet unsigned integer specifying an index into
@@ -499,13 +517,31 @@
		        <t hangText="occur(rence):">
	                  A four- or eight-octet UNIX time value specifying the
	                  time at which a leap second correction occurs.
+			  The first value, if present, MUST be nonnegative,
+			  and each later value MUST be at least 2419199
+			  greater than the previous value.
+			  (This is 28 days' worth of seconds, minus a
+			  potential negative leap second.)
	                </t>
		        <t hangText="corr(ection):">
-	                  A four-octet signed integer specifying the total
-	                  number of leap seconds to be applied to UTC on or
-	                  after the occurrence.
+	                  A four-octet signed integer specifying the value
+			  of TAI - UTC - 10 on or after the occurrence.
+			  The correction value in the first leap second record,
+			  if present, MUST be either one (1) or minus one (-1).
	                  The correction values in adjacent leap second
	                  records MUST differ by exactly one (1).
+			  For timestamps that occur before the occurrence time
+			  in the first leap second record (or for all
+			  timestamps if there are no leap second records), the
+			  value of TAI - UTC - 10 is zero, and this zero value
+			  applies even to timestamps before the introduction
+			  of TAI or UTC.
+			  (The expression "TAI - UTC - 10" comes from the fact
+			  that TAI - UTC was defined to be 10 just prior to
+			  the first leap second in 1972, so clocks with leap
+			  seconds that use the UNIX Time origin of 1970 have a
+			  zero offset relative to UTC before the first leap
+			  second.)
	                </t>
	              </list>
		    </t>
@@ -513,6 +549,7 @@
	              A series of one-octet values indicating whether
	              the transition times associated with local time types were
	              specified as standard time or wall clock time.
+		      Each value MUST be 0 or 1.
	              A value of one (1) indicates standard time,
	              and MUST be set to one (1) if the corresponding
	              UT/local indicator is set to one (1).
@@ -527,6 +564,7 @@
	              A series of one-octet values indicating whether
	              the transition times associated with local time types were
	              specified as UT or local time.
+		      Each value MUST be 0 or 1.
	              A value of one (1) indicates UT,
	              and the corresponding standard/wall indicator MUST also
	              be set to one (1).
@@ -567,9 +605,8 @@
	        "AKST9AKDT").</t>
	
	        <t>In order to eliminate unused space in a TZif file, every
-	        local time type record SHOULD be used by at least one
-	        transition (except that time type 0 need not be used if there
-	        is at least one transition).
+		nonzero local time type index SHOULD appear at least once in the
+		transition type array.
	        Likewise, every character in the time zone designations array
	        SHOULD be used by at least one time type record.</t>
	      </section>  <!-- data -->
@@ -601,8 +638,8 @@
	              of the "Base Definitions" Volume of <xref
	              target="POSIX"/>.  If empty, the corresponding
	              information is not available.
-	              If the string is nonempty, at least one transition MUST
-	              be present in the 64-bit data, and the TZ string MUST be
+		      If the string is nonempty and one or more transitions
+		      appear in the 64-bit data, the string MUST be
	              consistent with the last 64-bit transition - i.e.,
	              evaluating the TZ string at the time of the last
	              transition should yield the same time type as the time