Subject: [Tzdist-bis] annotated idnits for draft-murchison-tzdist-tzif-09

From: Eliot Lear <[email-masked]>
Date: Sun, 1 Jul 2018 16:12:46 +0200

Please see below:

>   Checking boilerplate required by RFC 5378 and the IETF Trust (see
>   https://trustee.ietf.org/license-info):
>   ----------------------------------------------------------------------------
>
>      No issues found here.

Goodie.

>
>   Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
>   ----------------------------------------------------------------------------
>
>      No issues found here.

Goodie.

>
>   Checking nits according to https://www.ietf.org/id-info/checklist :
>   ----------------------------------------------------------------------------
>
>   -- The draft header indicates that this document updates RFC7808, but the
>      abstract doesn't seem to mention this, which it should.

I don't think it does.

>
>
>   Miscellaneous warnings:
>   ----------------------------------------------------------------------------
>
>   == The document seems to lack the recommended RFC 2119 boilerplate, even if
>      it appears to use RFC 2119 keywords -- however, there's a paragraph with
>      a matching beginning. Boilerplate error?

Ignore this.

>
>      (The document does seem to have the reference to RFC 2119 which the
>      ID-Checklist requires).
>   == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD',
>      or 'RECOMMENDED' is not an accepted usage according to RFC 2119.  Please
>      use uppercase 'NOT' together with RFC 2119 keywords (if that is what you
>      mean).

Fix the following:

>      
>      Found 'MUST not' in this paragraph:
>      
>      utoff:  A four-octet signed integer specifying the number of
>      seconds to be added to UT in order to determine local time. The value
>      MUST NOT be -2**31, and SHOULD be in the range [-89999, 93599] (i.e., its
>      value SHOULD be more than -25 hours and less than 26 hours).  (Avoiding
>      -2**31 allows 32-bit clients to negate the value without overflow. 
>      Restricting it to [-89999, 93599] allows easy support by implementations
>      that already support the the POSIX-required range [-24:59:59, 25:59:59].)
OLD:

>       The value MUST not be -2**31, and SHOULD be in the range [-93599, 93599]
<<<

NEW:

>       The value MUST NOT be -2**31, and SHOULD be in the range [-93599, 93599]
<<<

>      (i.e., its absolute value SHOULD be less than 26 hours).
>
>
>   Checking references for intended status: Proposed Standard
>   ----------------------------------------------------------------------------
>
>      (See RFCs 3967 and 4897 for information about using normative references
>      to lower-maturity documents in RFCs)
>
>   -- Looks like a reference, but probably isn't: '1' on line 797
Ignore this.
>
>   == Missing Reference: '-89999' is mentioned on line 352, but not defined
Ignore this.
>
>   -- Looks like a reference, but probably isn't: '93599' on line 355
>
>   == Missing Reference: '-93599' is mentioned on line 355, but not defined
Ignore this.
>
>   -- Looks like a reference, but probably isn't: '2' on line 799
Ignore this.
>
>   -- Looks like a reference, but probably isn't: '3' on line 801
Ignore this.
>
>   -- Looks like a reference, but probably isn't: '4' on line 754
Ignore this.
>
>   -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX'

Ignore this.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Sun, 1 Jul 2018 15:37:33 +0100

>>       abstract doesn't seem to mention this, which it should.

The way in which it is used in conjunction with RFC7808 is perhaps not 
as clear as it might be? I still have not got my head around how one 
would download a complete set of data starting from a RFC7808 
identification of the source.

In addition, the changes to the definition of a time zone identifier 
format ( the restricted character set, etc. ) is perhaps an update to 
the broader definition in RFC7808?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Paul Eggert <[email-masked]>
Date: Sun, 1 Jul 2018 07:41:26 -0700

>>        The value MUST NOT be -2**31, and SHOULD be in the range [-93599, 93599]

Instead of doing that, let's delete the containing sentence, which is an editing 
error that nearly repeats the previous-but-one sentence and adds a good deal of 
confusion in the process. The attached patch fixes the editing error.
--- draft-murchison-tzdist-tzif-09.xml	2018-07-01 07:35:58.141960674 -0700
+++ draft-murchison-tzdist-tzif-09-1.xml	2018-07-01 07:36:20.935094128 -0700
@@ -455,10 +455,6 @@
			  Restricting it to [-89999, 93599] allows easy support by
			  implementations that already support the
			  the POSIX-required range [-24:59:59, 25:59:59].)
-	
-			  The value MUST not be -2**31, and SHOULD be in
-			  the range [-93599, 93599] (i.e., its absolute
-			  value SHOULD be less than 26 hours).
	                </t>
		        <t hangText="(is)dst:">
	                  A one-octet value indicating whether local

---

From: Eliot Lear <[email-masked]>
Date: Sun, 1 Jul 2018 16:48:43 +0200

Hi Lester,

> identification of the source.

I agree with this point.  The simplest way to address this would be to
note that this capability does not observe truncation rules, regardless
of whether the truncation capability is offered.  I don't know if that
is best.

> the broader definition in RFC7808?
As this is a new capability I'm not sure I understand your meaning.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Sun, 1 Jul 2018 16:05:40 +0100

> As this is a new capability I'm not sure I understand your meaning.

That is perhaps why I am having a problem with where this fits in. The 
binary data includes the time zone identifier, but the recent updates 
restrict the character set and format that may be used to define them. 
It may be that this is irrelevant if the time zone identifier provided 
by the RFC7808 lookup supersedes the one contained in the data? And some 
other mechanism identifies which block of binary data is actually 
accessed, perhaps via the alias facility in RFC7808? I don't see how 
this is a 'new capability', but rather just an alternate method of 
providing a binary package of data providing an individual rule set? In 
reality there is no reason to even access this capability if the 
identifier refers to a single fixed offset anyway?

---

From: Paul Eggert <[email-masked]>
Date: Sun, 1 Jul 2018 08:09:28 -0700

Lester Caine wrote:
> 
> The binary 
> data includes the time zone identifier

No, TZif binary data does not contain the time zone identifier (e.g., 
"America/Los_Angeles"). It contains only time zone abbreviations (e.g., "PDT"). 
Does this help resolve any confusion?

---

From: Lester Caine <[email-masked]>
Date: Sun, 1 Jul 2018 16:21:51 +0100

On 01/07/18 16:09, Paul Eggert wrote:
>> The binary data includes the time zone identifier
> 
> "America/Los_Angeles"). It contains only time zone abbreviations (e.g., 
> "PDT"). Does this help resolve any confusion?

That does now make more sense. I'll re-read things to check if that 
distinction is obvious ...

---

From: Paul Eggert <[email-masked]>
Date: Sun, 1 Jul 2018 11:04:03 -0700

Eliot Lear wrote:
> of whether the truncation capability is offered.

Would it suffice to prohibit TZDIST clients from asking for truncation in their 
"get" actions for TZif files? Please see attached patch for some proposed 
wording along these lines.

Also, to forestall possible confusion it'd be helpful to say explicitly that 
time zone identifiers and time zone metadata (in the RFC7808 sense) are out of 
scope for this spec. Please see attached patch for wording there too.

The patch also fixes a typo (missing ")") in this area.
--- draft-murchison-tzdist-tzif-09-2.xml	2018-07-01 07:52:31.084774198 -0700
+++ draft-murchison-tzdist-tzif-09-3.xml	2018-07-01 10:19:10.532275683 -0700
@@ -118,6 +118,12 @@
	      data or leap second information.
	      One such source is the IANA-hosted time zone database <xref
	      target="RFC6557" />.</t>
+	
+	      <t>This specification does not define the time zone identifier
+	      that names a TZif file, nor does it define time zone metadata
+	      such as publisher name, version identifier, aliases, or localized names.
+	      These are defined by the <xref target="RFC7808">Time Zone Data
+	      Distribution Service (TZDIST)</xref>.</t>
	    </section>  <!-- Intro -->
	
	    <section title='Conventions Used in This Document'>
@@ -757,7 +763,7 @@
	      that it supports this format by including the MIME media type
	      <xref target="tzif">"application/tzif"</xref> in its
	      "capabilities" response (see Section 5.1 of <xref
-	      target="RFC7808"/>.
+	      target="RFC7808"/>).
	      A TZDIST service MAY also include the MIME media type
	      <xref target="tzif-leap">"application/tzif-leap"</xref> in its
	      "capabilities" response if it is able to generate TZif files
@@ -768,7 +774,10 @@
	      <t>TZDIST clients MUST use the HTTP
	      <xref target="RFC7231">"Accept"</xref> header field to indicate
	      their preference to receive data in the "application/tzif"
-	      and/or "application/tzif-leap" formats.</t>
+	      and/or "application/tzif-leap" formats.
+	      TZDIST clients requesting TZif files MUST NOT use "get" actions
+	      with "start" or "end" URI query parameters, which request time
+	      zone data truncation (see <xref target='RFC7808'/> Section 5.3).</t>
	
	      <section title='Example'>
	        <t>In this example, the client checks the server for the

---

From: Lester Caine <[email-masked]>
Date: Sun, 1 Jul 2018 23:48:26 +0100

On 01/07/18 19:04, Paul Eggert wrote:
> Would it suffice to prohibit TZDIST clients from asking for truncation 
> in their "get" actions for TZif files? Please see attached patch for 
> some proposed wording along these lines.

But the whole point here is that the TZif data IS truncated and should 
NOT be presented as being otherwise. The TZDIST rules were intended to 
be honest in presenting data, and to provide data that does not actually 
return 'untruncated' material is simply wrong! The only valid 
information that TZif can provide is data that starts in 1970 ... as the 
published restrictions on TZ dictate ...

---

From: Paul Eggert <[email-masked]>
Date: Sun, 1 Jul 2018 16:23:29 -0700

Lester Caine wrote:
> But the whole point here is that the TZif data IS truncated

Version 2 and 3 data use signed 64-bit timestamps, and so are not truncated in 
any practical sense. True, version 1 timestamps are limited to signed 32-bit 
values, but version 1 is deprecated and the TZDIST truncation approach wouldn't 
work well with version 1 anyway; we shouldn't go down that route.

> The only valid information that TZif can provide is 
> data that starts in 1970 ... as the published restrictions on TZ dictate ...

No, the draft TZif spec requires support for negative timestamps as an extension 
to POSIX. The spec does not require data truncation due to the epoch in 1970, 
and TZif files in practice are not truncated in that way.

---

From: Lester Caine <[email-masked]>
Date: Mon, 2 Jul 2018 09:26:31 +0100

On 02/07/18 00:23, Paul Eggert wrote:
> Version 2 and 3 data use signed 64-bit timestamps, and so are not 
> truncated in any practical sense. True, version 1 timestamps are limited 
> to signed 32-bit values, but version 1 is deprecated and the TZDIST 
> truncation approach wouldn't work well with version 1 anyway; we 
> shouldn't go down that route.
> 
>> The only valid information that TZif can provide is data that starts 
>> in 1970 ... as the published restrictions on TZ dictate ...
> 
> No, the draft TZif spec requires support for negative timestamps as an 
> extension to POSIX. The spec does not require data truncation due to the 
> epoch in 1970, and TZif files in practice are not truncated in that way.

This is nothing to do with the mechanics of how you present the data, 
but the data itself! If I select 'Europe/Guernsey', 'Europe/IsleOfMan' 
or 'Europe/Jersey' I should be able to establish if the data supplied is 
correct prior to 1970, or simply a copy of 'Europe/London'! Untruncated 
data provided in RFC7808 relates to the period of time zones existence 
and a publisher providing truncated data only will also publish the 
period for which that data is valid! This was to allow publishers to 
only provide say the current 20 years of data if they preferred.

TZ data as a source is NOT un-truncated and therefore it's limitations 
should be published in TZDIST ... In the absence of any more accurate 
information, TZ as a source should be published with a truncation 
starting with 1970 and perhaps ending in 2038 but for some timezones, 
the published data IS accurate for a longer period. It is up to the 
TZDIST publisher to identify that where it is documented accurate data.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Jul 2018 11:07:57 +0200

Hi Lester,

> TZDIST publisher to identify that where it is documented accurate data.
I don't agree with this approach.  For one thing, 1970 is arbitrary in
this context.  2038 is similarly arbitrary for versions 2 and 3.  The
issue you are raising is based on the beginning of the existence of an
time zone identifier.  The standard requires only that contiguous data
be returned, and not any particular period of time for the identifier to
exist.  The TZ database itself attempts accuracy well before 1970.  The
only issue is when new identifiers are proposed to cover time prior to
that date.  That's not at issue here.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Mon, 2 Jul 2018 10:32:24 +0100

> that date.  That's not at issue here.

*NO* ... the TZ database deliberately hides some pre-1970 data in the 
'backzone' file and in recent years the truncation of data to 1970 has 
been carried out as the remit of the TZ database IS only from 1970. The 
issue is that TZif carries no payload that allows one to check if the 
binary data returned has accurate pre-1970 data for the time zone 
identifier being returned.

My point is that in the absence of more accurate truncation dates for 
each time zone rule set, the only thing we can be accurate about is the 
1970 to 2038 period. That TZ MAY have more accurate pre-1970 data is a 
fact, but there is no 'warrantee' on the data provided by TZif for the 
pre-1970 period and currently many distributions do not include the 
backzone file, or easily identify if the data supplied is the correct 
data or simply a post-1970 alias ...

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Jul 2018 13:36:30 +0200

I'll leave it to the TZ Database administrator to answer this question
authoritatively but..

On 02.07.18 11:32, Lester Caine wrote:
> been carried out as the remit of the TZ database IS only from 1970.
> The issue is that TZif carries no payload that allows one to check if
> the binary data returned has accurate pre-1970 data for the time zone

As you know, that file is not recommended to be used operationally.  Are
you suggesting you want an option for backzone data or would you like a
recommendation that backzone data not be included?

> each time zone rule set, the only thing we can be accurate about is
> the 1970 to 2038 period. That TZ MAY have more accurate pre-1970 data
> is a fact, but there is no 'warrantee' on the data provided by TZif
> for the pre-1970 period and currently many distributions do not
> include the backzone file, or easily identify if the data supplied is
> the correct data or simply a post-1970 alias ...

And so you are suggesting that truncation occur to the point where
backzone data ends (e.g.,  the time_t of the last backzone transition +1)?

Eliot

---

From: Lester Caine <[email-masked]>
Date: Mon, 2 Jul 2018 13:31:29 +0100

> recommendation that backzone data not be included?

The problem CURRENTLY is that we have no idea as to whether the 
additional verified material pushed out to the backzone file is included 
or not. More than likely it is not included, so managing historic 
material as recently as the 2nd World War can not assume that users are 
seeing the same time zone data. We have to assume that the time zone 
data is unreliable everywhere.

> backzone data ends (e.g.,  the time_t of the last backzone transition +1)?

That is a simplification but no I'm not suggesting that. A publishing 
source of TZDIST that is attempting to fill the gap between TZ and 
earlier historic material would sill not be advertising totally 
untruncated data. We do not have data for some areas of the world to 
fill in all the gaps, but where we do have a valid start date for a time 
zone, that would be the one provided. TZ could be massaged to provide a 
suitable date prior to 1970 where documented, and for the UK pre-1970 
time zones we do have that data stored in the backzone file, so if TZif 
is providing a view including backzone, we are in a better position than 
simply assuming the single pre-1970 rule set applies were it does not.

The assumption is that historic material is such a small user case that 
we don't need to worry about it. THAT has been said many times on the TZ 
list, but in that case the TZif should simply return 'don't know' where 
the information is not valid, and a start date flags that 'don't know' 
point and directs one to find an alternate source of offsets for that 
period.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 08:59:44 -0400

> some proposed wording along these lines.

I don't see why a TZDIST service can't return truncated TZif data. My 
service certainly can.

And with respect to the 1970 discussion, this draft only talks about the 
TZif binary format, NOT the source of the data.  Whether the IANA tzdata 
is truncated is out of scope.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Jul 2018 16:10:40 +0200

> I don't see why a TZDIST service can't return truncated TZif data. My

I would view that as an implementation pain.  Do you then want to fully
support the truncated capability in this draft?
>
> And with respect to the 1970 discussion, this draft only talks about
> the TZif binary format, NOT the source of the data.  Whether the IANA
> tzdata is truncated is out of scope.
>

Good point.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 10:27:53 -0400

On 07/02/2018 10:10 AM, Eliot Lear wrote:
> I would view that as an implementation pain.

Since TZif is mostly just a list of transitions, I don't see how its all 
that difficult, especially for a service that already supports 
truncation of iCalendar data.

> Do you then want to fully

I believe that any actions that the TZDIST service supports for 
text/calendar must also be supported for application/tzif.  I never even 
gave any consideration to this not being the case.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Jul 2018 16:28:56 +0200

Ok.  It's probably worth mentioning, then, that files should observe
truncation rules when the capability is offered.

On 02.07.18 16:27, Ken Murchison wrote:
> Since TZif is mostly just a list of transitions, I don't see how its
> all that difficult, especially for a service that already supports
> text/calendar must also be supported for application/tzif.  I never
> even gave any consideration to this not being the case.
>
>

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 12:57:41 -0400

> +	      Distribution Service (TZDIST)</xref>.</t>

Reworded the existing paragraph to incorporate this info.

I left out the patch saying that truncation of TZif files via TZDIST 
isn't allowed.  As suggested by Eliot, I added text to state that all 
features supported for other MIME types MUST be supported for 
"application/tzif".

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 11:29:11 -0700

Ken Murchison wrote:
> I left out the patch saying that truncation of TZif files via TZDIST isn't 
> allowed.  As suggested by Eliot, I added text to state that all features 
> supported for other MIME types MUST be supported for "application/tzif".

Off the top of my head I see three real problems with this.

First, how a server can support truncation to arbitrary dates while also 
conforming to the requirement that tzif-leap files MUST contain a nonzero number 
leap seconds? If a client asks for a tzif-leap file for America/Los_Angeles 
truncated to 2017-2020, the client presumably wants a TZif file with zero leap 
seconds, because there haven't been any leap seconds since 2017. But that would 
violate a MUST.

Second, even if the client asks for tzif-leap America/Los_Angeles 2016-2020 and 
thus gets a leap second record for 2016-12-31 23:59:60 UTC, this leap second 
record won't have 'corr' value of 1, and that would violate another MUST.

Third, I don't see how truncation works together with the spec's rules about 
timestamps before the first transition, or timestamps after the last one. If a 
server omits transitions from TZif data, it produces a TZif file that stills 
specifies UTC offsets for timestamps from minus to plus infinity. I can sort of 
see how that might work (I guess you're supposed never use the file to interpret 
timestamps outside the requested range), but the spec contains no guidance here 
and guidance is clearly needed. For example, if the server truncates to 
timestamps from 2010 through 2020, should the server be required to omit the 
POSIX-like TZ string at the end, since the file shouldn't be specifying data 
after 2020? Or suppose the server advertises truncation but simply ignores 
requests to truncate data (always generating the full TZif file); is that OK?

These are the sorts of considerations that went through my mind when I proposed 
text saying that truncation is not allowed. If we're going to allow truncation, 
we should at least put in weasel words saying something like "This specification 
does not define how truncation works, or how to interpret truncated TZif files."

Better, of course, would be to specify the details that are missing in the spec 
now. Ken, since you've implemented truncation, you're probably the best one to 
write up those details.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 11:51:01 -0700

Lester Caine wrote:
> TZif carries no payload that allows one to check if the binary 
> data returned has accurate pre-1970 data for the time zone identifier being 
> returned.

It's true that the TZif format contains no payload about the name of the file, 
or its version number, or its provenance, or whether its data entries are 
truncated, or any other metadata such as how many known discrepancies are 
present in the data. But TZif is what it is, and we can't change it to address 
these issues in this spec. It's also too much to ask TZDIST to supply metadata 
that is absent from TZif and is not recorded systematically anywhere. So it's 
pretty clear that we should let this sleeping dog lie.

I'm not saying that there's not an issue here; far from it. It's just that it's 
out of scope for this spec. Maybe we could address it in a future version, but 
not now.

---

From: Lester Caine <[email-masked]>
Date: Mon, 2 Jul 2018 20:53:04 +0100

On 02/07/18 19:51, Paul Eggert wrote:
> Lester Caine wrote:
>> TZif carries no payload that allows one to check if the binary data 
>> returned has accurate pre-1970 data for the time zone identifier being 
>> returned.
> 
> It's true that the TZif format contains no payload about the name of the 
> file, or its version number, or its provenance, or whether its data 
> entries are truncated, or any other metadata such as how many known 
> discrepancies are present in the data. But TZif is what it is, and we 
> can't change it to address these issues in this spec. It's also too much 
> to ask TZDIST to supply metadata that is absent from TZif and is not 
> recorded systematically anywhere. So it's pretty clear that we should 
> let this sleeping dog lie.

But by NOT documenting the limitations of the current TZif people will 
not be aware that there is a problem? This sleeping dog has existed for 
so long now it needs waking up a bit to highlight that TZ does have 
known discrepancies. Either by design due to backzone, or because we do 
not have all the data.

> I'm not saying that there's not an issue here; far from it. It's just 
> that it's out of scope for this spec. Maybe we could address it in a 
> future version, but not now.

Bringing it back in scope ... If I ask for a TZif record for 
'Europe/IsleOfMan', what will be served up? A duplicate copy of 
'Europe/London'? Is there no way that the duplicate truncated copy can't 
be flagged in this instance ... along with Guernsey and Jersey ... and 
the other truncated aliases.

I presume if I create my own server which ensures backzone is loaded, 
then I will be serving the correct data, but in this case 
'Europe/IsleOfMan' will not match the data provided by a non-backzone 
server source with NOTHING to say that this is the case? Ignoring 
anything else that may be out of scope, this is very much in scope of 
providing matching data to a user? Or is there no requirement that TZif 
packets served up actually match one another?

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 15:59:12 -0400

On 07/02/2018 02:29 PM, Paul Eggert wrote:
> First, how a server can support truncation to arbitrary dates while 
> also conforming to the requirement that tzif-leap files MUST contain a 
> nonzero number leap seconds? If a client asks for a tzif-leap file for 
> America/Los_Angeles truncated to 2017-2020, the client presumably 
> wants a TZif file with zero leap seconds, because there haven't been 
> any leap seconds since 2017. But that would violate a MUST.
>
> Second, even if the client asks for tzif-leap America/Los_Angeles 
> 2016-2020 and thus gets a leap second record for 2016-12-31 23:59:60 
> UTC, this leap second record won't have 'corr' value of 1, and that 
> would violate another MUST.

I think any leap second records that contribute to the transition times 
need to be in the file.  So, for you examples above all leap seconds 
would be included.  If a client asked for 1950-1960, then no leap second 
records would appear.  For 2000-2010, leap second records for 2011 and 
later could be omitted.

>
> Third, I don't see how truncation works together with the spec's rules 
> about timestamps before the first transition, or timestamps after the 
> last one. If a server omits transitions from TZif data, it produces a 
> TZif file that stills specifies UTC offsets for timestamps from minus 
> to plus infinity. I can sort of see how that might work (I guess 
> you're supposed never use the file to interpret timestamps outside the 
> requested range), but the spec contains no guidance here and guidance 
> is clearly needed. For example, if the server truncates to timestamps 
> from 2010 through 2020, should the server be required to omit the 
> POSIX-like TZ string at the end, since the file shouldn't be 
> specifying data after 2020? Or suppose the server advertises 
> truncation but simply ignores requests to truncate data (always 
> generating the full TZif file); is that OK?

The same methodology that is described in RFC 7808 for truncating 
iCalendar data would apply.  In fact my server generates TZif from 
iCalendar.  That said, we don't have a way in TZif to specify the end 
truncation time like we can with the TZUNTIL property in iCalendar.  I 
need to think about this.

I can try to come up with some text for handling truncation.  I assume 
that there is no issue with TZDIST "expand" with TZif, since for the 
most part the transitions are already expanded.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Mon, 2 Jul 2018 16:12:06 -0400

Hi Lester,

On 07/02/2018 03:53 PM, Lester Caine wrote:
>>> returned has accurate pre-1970 data for the time zone identifier 
>>> being returned.
>>
>> It's true that the TZif format contains no payload about the name of 
>> the file, or its version number, or its provenance, or whether its 
>> data entries are truncated, or any other metadata such as how many 
>> known discrepancies are present in the data. But TZif is what it is, 
>> and we can't change it to address these issues in this spec. It's 
>> also too much to ask TZDIST to supply metadata that is absent from 
>> TZif and is not recorded systematically anywhere. So it's pretty 
>> clear that we should let this sleeping dog lie.
>
> not be aware that there is a problem? This sleeping dog has existed 
> for so long now it needs waking up a bit to highlight that TZ does 
> have known discrepancies. Either by design due to backzone, or because 
> we do not have all the data.
>
> 'Europe/London'? Is there no way that the duplicate truncated copy 
> can't be flagged in this instance ... along with Guernsey and Jersey 
> ... and the other truncated aliases.
>
> providing matching data to a user? Or is there no requirement that 
> TZif packets served up actually match one another?

You seem to be treating the TZif format and the IANA tzdata as 
equivalent.  This is simply not the case.  I can take time zone 
transitions from any source that I like and create a TZif file.  No 
different than creating iCalendar VTIMEZONEs.

The deficiency that you bring up did, and still exists, for a TZDIST 
server that returns iCalendar.  IMO, the historical/versioning issue 
probably needs to be addressed with a new extension to TZDIST first.  
Then perhaps new properties/parameters need to be added to iCalendar and 
a version 4 of TZif need to be specified to accommodate your use case.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 13:52:32 -0700

Ken Murchison wrote:

>> If a client asks for a tzif-leap file for 
>> America/Los_Angeles truncated to 2017-2020, the client presumably wants a TZif 
>> file with zero leap seconds, because there haven't been any leap seconds since 
>> 2017. But that would violate a MUST.
>>
>> Second, even if the client asks for tzif-leap America/Los_Angeles 2016-2020 
>> and thus gets a leap second record for 2016-12-31 23:59:60 UTC, this leap 
>> second record won't have 'corr' value of 1, and that would violate another MUST.
> 
> I think any leap second records that contribute to the transition times need to 
> be in the file.  So, for you examples above all leap seconds would be included.

What's the point of truncation if you must include leap second records that are 
not needed for the asked-for time range? For my first example, only one leap 
second record is needed, to establish TAI-UTC for 2017-2020. For the second one, 
only two records are needed, one for 2016 and the other for 2017-2020.

> If a client asked for 1950-1960, then no leap second records would appear.

That would violate a MUST requirement, no? The draft says that for 
application/tzif-leap, 'leapcnt' MUST be nonzero. So this part of the draft 
would need to be rethought somehow.

> The same methodology that is described in RFC 7808 for truncating iCalendar data 
> would apply.  In fact my server generates TZif from iCalendar.  That said, we 
> don't have a way in TZif to specify the end truncation time like we can with the 
> TZUNTIL property in iCalendar.  I need to think about this.

Yes, I'm afraid that it's not as obvious to me as it is to you. No doubt this is 
because I don't know iCalendar well. While you're thinking about it, you might 
consider writing up something that would cover issues like the following:

* Truncation can be tricky, because neither TZif nor VTIMEZONE dominates the 
other. They both can have info that the other format can't represent.

* For the TZUNTIL problem, I suggest saying that the truncated TZif file must 
accurately represent all timestamps in the requested range. The TZif file need 
not represent the TZUNTIL value itself, since this can't be done efficiently in 
general, in TZif format.

* I assume the truncation rules should apply directly only to the 64-bit TZif 
data, and that we don't worry about truncation of 32-bit data. Otherwise things 
will get pretty hairy and we don't want to waste our time worrying about 
applying this new feature (truncation) to an obsolete format (version 1). For 
example,iCalendar requires that there must be a DTSTART at the start point of 
the truncation range, but that's not possible for pre-1901 start points in 
version 1 format.

> I assume that 
> there is no issue with TZDIST "expand" with TZif, since for the most part the 
> transitions are already expanded.

Unfortunately that assumption isn't quite correct, as the "expand" question 
comes up when expanding the POSIX-like TZ string into transitions in the TZif 
data blocks. zic expands the data to 2038, to cater to older clients that don't 
grok TZ strings. Is this zic behavior allowed when truncation is requested?

All in all, I think our best bet is to not specify detailed truncation rules, 
and merely to say that a TZif file truncated to a range R must correctly 
represent all timestamps in R. That's simple and easy to write down and 
understand, and it captures the intent of truncation.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 13:55:59 -0700

Paul Eggert wrote:
> 
> neither TZif nor VTIMEZONE dominates the 

This point is worth putting directly into the spec, as part of the introduction. 
Proposed patch attached.
--- draft-murchison-tzdist-tzif-09-4.xml	2018-07-01 11:40:11.878743855 -0700
+++ draft-murchison-tzdist-tzif-09-5.xml	2018-07-02 13:29:05.121756694 -0700
@@ -112,7 +112,10 @@
	      time.  There is a wide variety of interoperable <xref
	      target="tz-link">software</xref> capable of generating and
	      reading files in this format.
-	      </t>
+	      TZif conveys information similar to that conveyed by the
+	      VTIMEZONE iCalendar component, although neither format dominates
+	      the other (that is, either format can contain information that
+	      cannot be represented in the other format).</t>
	
	      <t>This specification does not define the source of the time zone
	      data or leap second information.

---

From: Lester Caine <[email-masked]>
Date: Mon, 2 Jul 2018 22:20:47 +0100

> different than creating iCalendar VTIMEZONEs.

But where do you get your rule set data from? What would you provide if 
I asked for the VTIMEZONE for the Isle of Man?

What is needed going forward is a publisher that provides a more 
comprehensive set of data, and TZDIST as it stands can manage that, but 
needs a little help being able to return say 'TZ2017e' data in any form 
if the data being reviewed is tagged as being normalized via that 
version and one needs to update that earlier calendar. This is not 
'historic' material, this is current calendars, and systems can have 
several years of forward data that can be corrupted by untracked changes 
to offsets.

It is the CURRENT changes to TZ that I don't think are visible via TZif, 
so a short notice DST change is not easy to pick up, and TZDIST REQUIRES 
knowledge of the version of TZif that will be returned in order to 
ensure a change of data can be flagged. Ken I would presume your own 
source data fills this hole a different way in which case the two 
different TZif records can be created, and the correct TZDIST flags 
raised? I do not believe this is possible with IANA tzdata and is being 
flagged as 'out of scope', but surely one is useless without the other?

The scope may simply be to return a tzdata compatible binary file, with 
nothing to identify it's provenance, with information such as it's 
source and other meta data available only in TZDIST and simply stating 
that 'TZif does not define the source of data' sidesteps the problem of 
the downstream software even being aware that the information has 
changed? The vast majority of legacy software that does use time zone 
data is broken when a new version of TZ is loaded on the OS, and 
perpetuating that mistake here should be avoided! While the current 
software has no idea it's using 'TZ2017e' or what ever, isn't it time to 
include some element of version control now rather than perhaps getting 
around to it later with a new RFC?

---

From: Paul Eggert <[email-masked]>
Date: Mon, 2 Jul 2018 15:05:32 -0700

Lester Caine wrote:
> isn't it time to include some element of version control

That's not the job of this TZif spec. This issue applies to non-TZif data 
distributed by TZDIST too, and if it needs to be addressed it should be 
addressed at the TZDIST level, not here. TZDIST clients who want version control 
can use ETags or timestamps or URIs or whatever else they like within TZDIST, 
and the TZif spec shouldn't need to worry about the details.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 3 Jul 2018 12:55:37 +0200

On 02.07.18 22:52, Paul Eggert wrote:
>
> All in all, I think our best bet is to not specify detailed truncation
> rules, and merely to say that a TZif file truncated to a range R must
> correctly represent all timestamps in R. That's simple and easy to
> write down and understand, and it captures the intent of truncation. 
My concern with this is that it seems to concede that truncation is
indeed tricky.  If it really is THAT tricky, perhaps this is NOT the
correct format to use truncation with leap seconds.  Or, put another
way, perhaps we state that clients requesting application/tzif-leap MUST
NOT also specify a truncation range.

If we DO want to permit truncation ranges, then it seem as though the
following rules apply:

  * leapcnt MAY be zero for a given range.
  * It is presumed that implementations requesting data with leap
    seconds within a truncation range are either sufficiently aware of
    leap seconds that occurred PRIOR to the truncation period, or do not
    care.
  * The truncated TZif file must accurately represent all timestamps in
    the requested range. The TZif file need not represent the TZUNTIL
    value itself, since this can't be done efficiently in general, in
    TZif format

By the way, what does the following statement mean:

> The correction values in adjacent leap second records MUST differ by
> exactly one (1).

Does it mean that the correction value of an adjacent leap second record
MUST NOT equal the previous?  What happens if there are two leap seconds
in a row with the same adjustment?

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Tue, 3 Jul 2018 08:33:29 -0400

On 07/03/2018 06:55 AM, Eliot Lear wrote:
> Does it mean that the correction value of an adjacent leap second 
> record MUST NOT equal the previous?  What happens if there are two 
> leap seconds in a row with the same adjustment?
>

The correction values are a running total of leap seconds.  So for each 
occurrence the total goes up or down by one.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Tue, 3 Jul 2018 14:44:12 +0200

On 03.07.18 14:33, Ken Murchison wrote:
> The correction values are a running total of leap seconds.  So for
> each occurrence the total goes up or down by one.

Ok, got it.

Eliot

---

From: Paul Eggert <[email-masked]>
Date: Tue, 3 Jul 2018 09:57:29 -0700

> If we DO want to permit truncation ranges, then it seem as though the 

Yes, and there are also issues for what we do for "expand" requests.

I took a stab at specifying something along the direction that I think 
we're heading, which would permit truncation ranges; please see the 
attached proposed patch. However, I'm not an iCalendar expert and so may 
well have missed something.

--- draft-murchison-tzdist-tzif-09-6a.xml	2018-07-03 08:44:30.887368576 -0700
+++ draft-murchison-tzdist-tzif-09-7.xml	2018-07-03 09:53:22.723589147 -0700
@@ -534,7 +534,9 @@
		          A four-octet signed integer specifying the value
			  of TAI - UTC - 10 on or after the occurrence.
			  The correction value in the first leap second record,
-			  if present, MUST be either one (1) or minus one (-1).
+			  if present, MUST be either one (1) or minus one (-1)
+			  unless the TZif file has been truncated as described
+			  <xref target='tzdist'>below</xref>.</t>
	                  The correction values in adjacent leap second
	                  records MUST differ by exactly one (1).
			  For timestamps that occur before the occurrence time
@@ -752,7 +754,9 @@
	          to indicate that leap second records are included in the
	          TZif data;
	          'leapcnt' in the 64-bit header (or in the 32-bit
-	          header, if a version 1 file) MUST be non-zero.</t>
+	          header, if a version 1 file) MUST be non-zero
+		  unless the TZif file has been truncated as described
+		  <xref target='tzdist'>below</xref>.</t>
	
	          <t><xref target="tzif">"application/tzif"</xref>
	          to indicate that leap second records are not included in the
@@ -785,6 +789,34 @@
	      A TZDIST service MUST NOT advertise the "application/tzif-leap"
	      MIME media type without also advertising "application/tzif".</t>
	
+	      <t>When supplying time zone data that has been truncated
+	      to a time range (see <xref target='RFC7808'/> Section 5.3),
+	      a TZDIST service MUST supply a TZif file that represents
+	      the same timestamps within the truncation time range
+	      as the timestamps represented by the untruncated TZif file.
+	      (The truncated response can be byte-for-byte equivalent to the
+	      untruncated response, though this will likely be less efficient.)
+	      TZDIST clients SHOULD NOT use the truncated response
+	      to interpret timestamps out of the truncation time range.
+	      Although <xref target='RFC7808'/> Section 3.9 requires
+	      that a truncated "VTIMEZONE" iCalendar component must include
+	      "DTSTART" and/or "TZUNTIL" property values in some cases, there is no
+	      such requirement for TZif files since these iCalendar
+	      properties do not always have an efficient TZif representation.</t>
+	
+	      <t>When supplying TZif version 2 (or later) time zone data
+	      that has been expanded within a time range
+	      (see <xref target='RFC7808'/> Section 5.4),
+	      a TZDIST service MUST supply a TZif file that contains
+	      transition times in the 64-bit data block
+	      for all transitions within that time range.
+	      That is, the TZif file cannot rely on the POSIX-style TZ string
+	      to convey information about transitions within the time range.
+	      Although <xref target='RFC7808'/> Section 5.4 requires
+	      "start" and "end" members in the JSON response in some cases,
+	      there is no such requirement for TZif files since these
+	      members do not always have an efficient TZif representation.</t>
+	
	      <t>TZDIST clients MUST use the HTTP
	      <xref target="RFC7231">"Accept"</xref> header field to indicate
	      their preference to receive data in the "application/tzif"

---

From: Eliot Lear <[email-masked]>
Date: Tue, 10 Jul 2018 14:39:30 +0200

Ok... where are we this?  How do people feel about Paul's latest patch?

On 03.07.18 18:57, Paul Eggert wrote:
> attached proposed patch. However, I'm not an iCalendar expert and so
> may well have missed something.
>

---

From: Ken Murchison <[email-masked]>
Date: Tue, 10 Jul 2018 08:52:47 -0400

I took a break from editing due to the IETF blackout on submissions, but 
I think Paul's text is pretty close on truncation.  I *think* I need to 
add some text regarding time type 0 showing the offset prior to 
truncation.  I'm also wondering if leaving the TZ string empty would 
help signify TZUNTIL.

I don't see any issues with the TZDIST expand action and TZif, since its 
basically zdump with a different output format.  In fact, my server can 
spit out zdump format, which I have used for making sure that my version 
of vzic actually creates VTIMEZONEs that have transitions matching those 
of zic-produced TZif.

Also, I started thinking of the best way to combine the interoperability 
and implementation sections.  Its seems weird to have these separate.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Eliot Lear <[email-masked]>
Date: Tue, 10 Jul 2018 14:58:53 +0200

Hi Ken,

On 10.07.18 14:52, Ken Murchison wrote:
> I took a break from editing due to the IETF blackout on submissions,
> but I think Paul's text is pretty close on truncation.  I *think* I
> need to add some text regarding time type 0 showing the offset prior
> to truncation.  I'm also wondering if leaving the TZ string empty
> would help signify TZUNTIL.

Sorry- time type 0?  I guess we do need text ;-). I missed this one. 
What is it?

>
> I don't see any issues with the TZDIST expand action and TZif, since
> its basically zdump with a different output format.  In fact, my
> server can spit out zdump format, which I have used for making sure
> that my version of vzic actually creates VTIMEZONEs that have
> transitions matching those of zic-produced TZif.

Ok
>
> Also, I started thinking of the best way to combine the
> interoperability and implementation sections.  Its seems weird to have
> these separate.
>
Yes.

Eliot

---

From: Ken Murchison <[email-masked]>
Date: Tue, 10 Jul 2018 09:03:03 -0400

> Sorry- time type 0?  I guess we do need text ;-). I missed this one.

 From -09:

"Local time for timestamps before the first transition is specified by 
the first time type (time type 0)."

I was going to emphasize this w.r.t. truncation.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Tue, 10 Jul 2018 09:24:53 -0700

On 07/10/2018 05:52 AM, Ken Murchison wrote:
> I *think* I need to add some text regarding time type 0 showing the 
> offset prior to truncation.

Is the idea that time type 0's UT offset should equal what would be 
TZOFFSETFROM in the corresponding DTSTART property value for iCalendar, 
if iCalendar were truncated? Yes, that would make sense (I missed that, 
as I'm not an iCalendar guru). It could be a bit more complicated than 
iCalendar, though, as it's not just the UT offset; it's also the DST 
flag and the time zone abbreviation for type 0.

> I'm also wondering if leaving the TZ string empty would help signify 
> TZUNTIL.

That would not be good, because TZif has no other compact representation 
for a repeated rule. Suppose someone asks for truncation to the year 
range 1970-9999 but does not ask for expansion. We don't want to require 
the server to output a TZif file with an empty TZ string, as that would 
mean the file would have to contain explicit transitions for each year 
2019 through 9999, and this would be longer than the untruncated TZif 
file. For efficiency the server should be able to use a TZ string in a 
truncated file, just as it can use it in an untruncated one.

More generally, the TZif format wasn't designed for TZDIST truncation, 
and we shouldn't try to retroactively impose iCalendar truncation needs 
into TZif version 3. Version 4, if it's ever designed, would be a better 
candidate for this. In the meantime, all we should say is that a 
truncated TZif file correctly represents all timestamps in the truncated 
period.

> its basically zdump with a different output format.

Do you mean that my proposed changes for "expand" are OK, or that the 
draft is already fine for "expand" and doesn't need the proposed changes?

> interoperability and implementation sections.  Its seems weird to have 

Yes, it's odd to me too.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 10 Jul 2018 18:34:51 +0200

On 10.07.18 18:24, Paul Eggert wrote:
> and we shouldn't try to retroactively impose iCalendar truncation
> needs into TZif version 3. Version 4, if it's ever designed, would be
> a better candidate for this. In the meantime, all we should say is
> that a truncated TZif file correctly represents all timestamps in the
> truncated period.

The format of the file is so compact as compared to VTIMEZONE entries
that I really wonder what the utility of truncating at all really is,
especially with regard to files with leap seconds.  I understand
Lester's point, but IMHO if someone wants that truncated form then we
*have* VTIMEZONEs just for that.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 10 Jul 2018 13:18:25 -0400

On 07/10/2018 12:24 PM, Paul Eggert wrote:
> TZOFFSETFROM in the corresponding DTSTART property value for 
> iCalendar, if iCalendar were truncated? Yes, that would make sense (I 
> missed that, as I'm not an iCalendar guru). 

Correct.

> It could be a bit more complicated than iCalendar, though, as it's not 
> just the UT offset; it's also the DST flag and the time zone 
> abbreviation for type 0.

That info is also incorporated in iCalendar via the TZOFFSETFROM and 
TZNAME properties being contaianed in either a DAYLIGHT or STANDARD 
component.

> That would not be good, because TZif has no other compact 
> representation for a repeated rule. Suppose someone asks for 
> truncation to the year range 1970-9999 but does not ask for expansion. 
> We don't want to require the server to output a TZif file with an 
> empty TZ string, as that would mean the file would have to contain 
> explicit transitions for each year 2019 through 9999, and this would 
> be longer than the untruncated TZif file. For efficiency the server 
> should be able to use a TZ string in a truncated file, just as it can 
> use it in an untruncated one.

Fair enough.  At this point, I'm going ask the TZDIST authors if they 
have any opinion/guidance on whether we need/prohibit truncation of TZif.

> draft is already fine for "expand" and doesn't need the proposed changes?

I don't think we need to discuss "expand" in our draft.  "Expand" works 
on whatever internal representation of time zone data the TZDIST server 
has, and always outputs it in JSON format, unlike the "get" action which 
outputs (optionally truncated) iCalendar or TZif.

"Expand" is mostly for lightweight clients that don't want to, or might 
not be able to, expand iCalendar recurrence rules (or POSIX TZ strings) 
into a set of transitions for themselves.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Ken Murchison <[email-masked]>
Date: Tue, 10 Jul 2018 13:29:14 -0400

On 07/10/2018 01:18 PM, Ken Murchison wrote:
>> It could be a bit more complicated than iCalendar, though, as it's 
>> not just the UT offset; it's also the DST flag and the time zone 
>> abbreviation for type 0.
>

Actually, that's not true.  Only TZOFFSETFROM carries info about right 
before the start of the truncation range.  In the case of TZif time type 
0 would contain additional info as you mention.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC

---

From: Paul Eggert <[email-masked]>
Date: Tue, 10 Jul 2018 10:36:47 -0700

On 07/10/2018 10:18 AM, Ken Murchison wrote:
> I don't think we need to discuss "expand" in our draft.  "Expand" 
> works on whatever internal representation of time zone data the TZDIST 
> server has, and always outputs it in JSON format, unlike the "get" 
> action which outputs (optionally truncated) iCalendar or TZif.
>
> "Expand" is mostly for lightweight clients that don't want to, or 
> might not be able to, expand iCalendar recurrence rules (or POSIX TZ 
> strings) into a set of transitions for themselves. 

Thanks for explaining.

A lightweight client that can't expand POSIX TZ strings will be out of 
luck then, right? Since "expand" doesn't apply to the TZif format that 
contains POSIX TZ strings. It might be worth mentioning this somewhere.

---

From: Ken Murchison <[email-masked]>
Date: Tue, 10 Jul 2018 13:45:00 -0400

On 07/10/2018 01:36 PM, Paul Eggert wrote:
>> works on whatever internal representation of time zone data the 
>> TZDIST server has, and always outputs it in JSON format, unlike the 
>> "get" action which outputs (optionally truncated) iCalendar or TZif.
>>

No.  A TZif server should have no problem generating proper "expand" 
output from TZif.

-- 
Ken Murchison
Cyrus Development Team
FastMail US LLC