Subject: Re: [Tzdist-bis] comments on tzdist-tzif-08

From: Eliot Lear <[email-masked]>
Date: Sun, 1 Jul 2018 10:38:03 +0200

On 01.07.18 02:43, Paul Eggert wrote:
> Ken Murchison wrote:
>
>> - Since we now use the term 'octets' for the contents of time zone
>> designations, 'charcnt' seems out of place.  Should we change to
>> 'octetcnt'?
>
> In practice they're single-byte ASCII characters, which are both
> octets and characters. Though I don't feel about it strongly, I'd
> leave it alone since it's been called 'charcnt' since the 1980s.
>

Yes please.  There's a lot of code that uses that variable name.

---

From: Eliot Lear <[email-masked]>
Date: Sun, 1 Jul 2018 10:56:55 +0200

Let's go through these just a bit:

>   o  Version 1 files are considered a legacy format and SHOULD NOT be
>       generated, as they do not support transition times after the year
>       2038.

I would suggest:

Files that contain ONLY version 1 information MUST NOT be generated,
as they do not support transition times after the year 2038.

Unless we know of great harm that would be caused by an oversized v1 file...

>
>    o  Implementations SHOULD generate a version 3 file if TZ string
>       extensions are necessary to accurately model transition times.
>       Otherwise, version 2 files SHOULD be generated.

Is there a lot of s/w out there that does understand v2 but does not
understand v3?  I ask this because this recommendation would seemingly
necessitate an update in the perhaps not-to-distant future.

>
>    o  The sequence of time changes defined by the 32-bit header and data
>       block SHOULD be a contiguous subsequence of the time changes
>       defined by the 64-bit header and data block, and by the footer.

Just a little bit of "why" would be useful here.

>
>    o  Time zone designations SHOULD consist of at least three (3) and no
>       more than six (6) ASCII characters from the set of alphanumerics,
>       '-', and '+'.

A little more "why" here please.

>
>    o  When reading a version 2 or 3 file, implementations SHOULD ignore
>       the 32-bit header and data block except for the purpose of
>       skipping over them.

If we're going this far we should go a little farther:

 o Implementations SHOULD calculate the total length of the v1 part of
the file and compare that against the actual file size, to determine if
a v2 or v3 block follows the v1 block.

>
>    o  When a TZif file is used in a MIME message entity it SHOULD be
>       indicated by one of the following media types:

s/SHOULD/MUST/

>       *  "application/tzif-leap" (Section 8.2) to indicate that leap
>          second records are included in the TZif data; 'leapcnt' in the
>          64-bit header (or in the 32-bit header, if a version 1 file)
>          MUST be non-zero.

>
>       *  "application/tzif" (Section 8.1) to indicate that leap second
>          records are not included in the TZif data; 'leapcnt' in the
>          header(s) MUST be zero (0).

Just to be clear: is there harm in having leap data in application/tzif
information?  This requirement otherwise requires that the
implementation either maintain two copies or dig through the file to
determine whether there are leap seconds.

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Sun, 1 Jul 2018 11:43:54 -0700

> Unless we know of great harm that would be caused by an oversized v1 file...

The last time I checked, Android still generated v1-only data (packaged inside 
its own format) for native apps. I suppose this is on the theory that Android 
devices currently being distributed are not intended to survive past 2038 
without software updates. I wouldn't be surprised to see other uses like that, 
so I'm not sure we want to outlaw v1 quite just yet.

> necessitate an update in the perhaps not-to-distant future.

It would be hard to take a census. Version 3 format came out in tzdb 2013e and 
tzdb's zic generates transitions until 2038 to keep version 2 (and 1) clients 
happy until 2038. Also, version 3 format is needed (and used) only for timezones 
in Israel, Fiji, Greenland, north Chile, and Easter Island. So even if there is 
a lot of v2-only software out there, users probably won't notice issues unless 
they're in one of these relatively-out-of-the-way places and they have post-2038 
timestamps.

> Just a little bit of "why" would be useful here.  

Thanks, I added some "why" in the attached proposed patch.

> A little more "why" here please.

I added some wording in the attached patch.

> a v2 or v3 block follows the v1 block.

Sure, but this should be done for the 64-bit header and data block too, no? 
Proposed wording in the attached patch.

>>        indicated by one of the following media types:

That would disallow sending a TZif file as application/octet-stream, no? I doubt 
whether we want to go that far.

> information?

Yes there is. It would cause ordinary clients in GNU/Linux, macOS, etc. to 
observe leap seconds, contrary to POSIX.
--- draft-murchison-tzdist-tzif-09-3.xml	2018-07-01 10:19:10.532275683 -0700
+++ draft-murchison-tzdist-tzif-09-4.xml	2018-07-01 11:40:11.878743855 -0700
@@ -720,16 +720,27 @@
	        <t>The sequence of time changes defined by the 32-bit
	        header and data block SHOULD be a contiguous subsequence
	        of the time changes defined by the 64-bit header and data
-	        block, and by the footer.</t>
+	        block, and by the footer.
+	        This guideline helps obsolescent version 1 readers
+		agree with current readers about timestamps within the
+		contiguous subsequence.  It also lets writers not
+		catering to obsolescent readers use a 'timecnt' of zero
+		in the 32-bit data to save space.</t>
	
	        <t>Time zone designations SHOULD consist of at least three (3)
	        and no more than six (6) ASCII characters from the set of
-	        alphanumerics, '-', and '+'.</t>
+	        alphanumerics, '-', and '+'.
+	        This is for compatibility with POSIX requirements for
+	        time zone abbreviations.</t>
	
	        <t>When reading a version 2 or 3 file, implementations
	        SHOULD ignore the 32-bit header and data block except for
	        the purpose of skipping over them.</t>
	
+		<t>Implementations SHOULD calculate the total lengths of the
+		32- and 64-bit headers and data blocks and compare them against
+		the actual file size, as part of a validity check for the file.</t>
+	
	        <t>When a TZif file is used in a MIME message entity it SHOULD
	        be indicated by one of the following media types:

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 12:32:20 -0400

Patch applied.

On 07/01/2018 02:43 PM, Paul Eggert wrote:
> The last time I checked, Android still generated v1-only data 
> (packaged inside its own format) for native apps. I suppose this is on 
> the theory that Android devices currently being distributed are not 
> intended to survive past 2038 without software updates. I wouldn't be 
> surprised to see other uses like that, so I'm not sure we want to 
> outlaw v1 quite just yet.
>
>
> It would be hard to take a census. Version 3 format came out in tzdb 
> 2013e and tzdb's zic generates transitions until 2038 to keep version 
> 2 (and 1) clients happy until 2038. Also, version 3 format is needed 
> (and used) only for timezones in Israel, Fiji, Greenland, north Chile, 
> and Easter Island. So even if there is a lot of v2-only software out 
> there, users probably won't notice issues unless they're in one of 
> these relatively-out-of-the-way places and they have post-2038 
> timestamps.
>
>
> Sure, but this should be done for the 64-bit header and data block 
> too, no? Proposed wording in the attached patch.
>
>
> That would disallow sending a TZif file as application/octet-stream, 
> no? I doubt whether we want to go that far.
>
>
> Yes there is. It would cause ordinary clients in GNU/Linux, macOS, 
> etc. to observe leap seconds, contrary to POSIX.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 13:06:57 -0400

On 06/30/2018 08:43 PM, Paul Eggert wrote:
>> - Your interop section seems to describe mostly bugs in readers. I'm 
>> not sure we need/want a list of where non-compliant applications fail 
>> (unless we put it in an appendix?)  Do we have best practices to 
>> avoid some of these things?
>
> An appendix or other such section would be fine. By my count only five 
> of the bulleted issues are clear reader bugs; the other eight issues 
> describe what readers following older, vaguer specs will do when given 
> files that meet this newer spec. The first three bullet points are in 
> the latter category, as are the points about '<' and '>', the two 
> non-ASCII issues, weird time zone abbreviations, and negative DST.
>
> The idea is to document common problems in implementing and upgrading 
> the spec, to help TZif writers output files that avoid common pitfalls 
> in older or buggy TZif readers (this is the "be conservative in what 
> you do" part of Postel's law), and to help future spec-writers 
> (assuming there are any) see what sort of things they are up against 
> when they're defining version 4, 5, etc.

Would you want to rewrite these to be recommendations for a generator 
rather than a list of problems with older software?

Eliot, do you have any thoughts on how best to incorporate this info.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 13:11:05 -0400

On 07/02/2018 01:06 PM, Ken Murchison wrote:
>>> not sure we need/want a list of where non-compliant applications 
>>> fail (unless we put it in an appendix?)  Do we have best practices 
>>> to avoid some of these things?
>>
>> An appendix or other such section would be fine. By my count only 
>> five of the bulleted issues are clear reader bugs; the other eight 
>> issues describe what readers following older, vaguer specs will do 
>> when given files that meet this newer spec. The first three bullet 
>> points are in the latter category, as are the points about '<' and 
>> '>', the two non-ASCII issues, weird time zone abbreviations, and 
>> negative DST.
>>
>> the spec, to help TZif writers output files that avoid common 
>> pitfalls in older or buggy TZif readers (this is the "be conservative 
>> in what you do" part of Postel's law), and to help future 
>> spec-writers (assuming there are any) see what sort of things they 
>> are up against when they're defining version 4, 5, etc.
>

And if there isn't a recommended work-around, is there any point in 
listing the problem?

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Jul 2018 19:42:18 +0200

On 02.07.18 19:11, Ken Murchison wrote:
>
>>>
>>> The idea is to document common problems in implementing and
>>> upgrading the spec, to help TZif writers output files that avoid
>>> common pitfalls in older or buggy TZif readers (this is the "be
>>> conservative in what you do" part of Postel's law), and to help
>>> future spec-writers (assuming there are any) see what sort of things
>>> they are up against when they're defining version 4, 5, etc.
>>

I think it's perfectly fine to have a section called "Implementation
Guidance" that discusses all of this.  I would add that it is often
usefulshow an example file, encoded somehow (perhaps tcpdump-style or od
output or some such).

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 16:31:48 -0700

> usefulshow an example file, encoded somehow (perhaps tcpdump-style or od

OK, a proposed patch is attached to do that. Although it has examples, it 
doesn't have full-flown example files, as they'd be a pain to read (and to 
write...) and I'm hoping the inline examples are good enough.

This patch also attempts to separate problems in upgrading the spec (i.e., 
compatibility issues) from ordinary bugs in readers, by having two lists of 
problems.

The patch also mentions suggested workarounds.
--- draft-murchison-tzdist-tzif-09-5.xml	2018-07-02 13:29:05.121756694 -0700
+++ draft-murchison-tzdist-tzif-09-6.xml	2018-07-02 16:24:17.704087095 -0700
@@ -854,6 +854,144 @@
	      </section>
	    </section> <!-- TZDIST -->
	
+	    <section title='Implementation Guidance' anchor='guidance'>
+	
+	      <t>This section documents common problems in implementing this
+	      specification.
+	      Most of these are problems in generating TZif files
+	      used by readers conforming to predecessors of this specification.
+	      The goals of this section are (1) to help TZif writers output files
+	      that avoid common pitfalls in older or buggy TZif readers,
+	      (2) to help TZif readers avoid common pitfalls when reading files
+	      generated by future TZif writers, and
+	      (3) to help any future specification authors see what sort of problems
+	      arise when the TZif format is changed.</t>
+	
+	      <t>When new versions of the TZif format have been defined,
+	      a design goal has been that a reader can successfully use a TZif file
+	      even if the file is of a later TZif version than what the reader
+	      was designed for.
+	      When complete compatibility was not achieved, an attempt
+	      was made to limit glitches to rarely-used timestamps,
+	      and to allow simple partial workarounds in writers designed to
+	      generate new-version data useful even for older-version readers.
+	      This section attempts to document these compatibility issues
+	      and workarounds, as well as to document other common bugs in readers.</p>
+	
+	      <t>Interoperability problems with TZif include the following:
+	      <list style='symbols'>
+	    	<t>Some readers examine only 32-bit data.
+	        As a partial workaround,
+		a writer can output as much 32-bit data as possible.
+		However, a reader should ignore 32-bit data, and should use 64-bit data
+		even if the reader's native timestamps have only 32 bits.</t>
+	
+		<t>Some readers designed for version 2 might mishandle timestamps
+		after a version 3 file's last transition, because they cannot
+		parse extensions to POSIX in the TZ-like string.
+		As a partial workaround,
+		a writer can output more transitions than necessary,
+		so that only far-future timestamps are mishandled
+		by version 2 readers.</t>
+	
+		<t>Some readers designed for version 2 do not support
+		permanent daylight saving time, e.g.,
+		a TZ string "EST5EDT,0/0,J365/25" denoting
+	        permanent Eastern Daylight Time (-04).
+		As a partial workaround,
+		a writer can substitute standard time for the next
+	        time zone east, e.g.,
+		"AST4" for permanent Atlantic Standard Time (-04).</t>
+	
+	    	<t>Some readers ignore the footer, and instead
+	    	predict future timestamps from the time type of the last transition.
+	        As a partial workaround,
+		a writer can output more transitions than necessary.</t>
+	
+	    	<t>Some readers do not use time type 0 for timestamps
+	    	before the first transition,
+		in that they infer a time type using a heuristic
+		that does not always select time type 0.
+	        As a partial workaround, a writer can output a dummy (no-op)
+		first transition at an early time.</t>
+	
+	    	<t>Some readers mishandle timestamps before the first transition
+	    	that has a timestamp not less than -2**31.
+	    	Readers that support only 32-bit timestamps are likely to be more
+	    	prone to this problem, for example,
+		when they process 64-bit transitions only some of which are
+		representable in 32 bits.
+	        As a partial workaround, a writer can output a dummy
+	        transition at timestamp -2**31.</t>
+	
+	    	<t>Some readers mishandle a transition if its
+	    	timestamp has the minimum possible signed 64-bit value.
+	        Timestamps less than -2**59 are not recommended.</t>
+	
+	    	<t>Some readers mishandle POSIX-style TZ strings that
+	    	contain '&lt;' or '&gt;'.
+	        As a partial workaround, a writer can avoid using '&lt;' or '&gt;'
+	        for time zone abbreviations containing
+		only alphabetic characters.</t>
+	
+	    	<t>Many readers mishandle time zone abbreviations that
+	    	contain non-ASCII characters.
+	        These characters are not recommended.</t>
+	
+	    	<t>Some readers may mishandle time zone abbreviations that
+	    	contain fewer than 3 or more than 6 characters, or that
+	    	contain ASCII characters other than alphanumerics, '-',
+	    	and '+'.
+	        These abbreviations are not recommended.</t>
+	
+	    	<t>Some readers mishandle TZif files that specify
+	    	daylight-saving time UT offsets that are less than the
+	    	UT offsets for the corresponding standard time.
+	    	These readers do not support locations like Ireland,
+	    	which uses the equivalent of the POSIX TZ string
+	    	"IST-1GMT0,M10.5.0,M3.5.0/1", observing standard time
+	    	(IST, +01) in summer and daylight saving time (GMT, +00)
+	    	in winter.
+	        As a partial workaround, a writer can output data for
+		the equivalent of the POSIX TZ string
+	        "GMT0IST,M3.5.0/1,M10.5.0",
+		thus swapping standard and daylight saving time.
+	        Although this workaround misidentifies which part of
+	        the year uses daylight saving time,
+	        it records UT offsets and time zone abbreviations correctly.</t>
+	      </list>
+	    </t>
+	
+	      <t>Some interoperability problems are reader bugs that
+	      are listed here mostly as warnings to developers of readers.</t>
+	      <list style='symbols'>
+		<t>Some readers do not support negative timestamps.
+		Developers of distributed applications should keep this
+		in mind if they need to deal with pre-1970 data.</t>
+	
+		<t>Some readers mishandle timestamps before the first transition
+		that has a nonnegative timestamp.
+		Readers that do not support negative timestamps are likely
+		to be more prone to this problem.</t>
+	
+	    	<t>Some readers mishandle time zone abbreviations like "-08" that
+	    	contain '+', '-', or digits.</t>
+	
+	    	<t>Some readers mishandle UT offsets that are out
+	    	of the traditional range of -12 through +12 hours, and
+	    	so do not support locations like Kiritimati that are
+	    	outside this range.</t>
+	
+	    	<t>Some readers mishandle UT offsets in the range [-3599, -1]
+	    	seconds from UT, because they integer-divide the offset by 3600
+	    	to get 0 and then display the hour part as "+00".</t>
+	
+	    	<t>Some readers mishandle UT offsets that are not
+	    	a multiple of one hour, or of 15 minutes, or of 1 minute.</t>
+	      </list>
+	    </t>
+	    </section>  <!-- guidance -->
+	
	    <section title='Security Considerations' anchor='security'>
	      <t>The Time Zone Information Format contains no executable
	      code and the format does not define any extensibility