Subject: [Tzdist] addressing issue 18: " removal of timezones?"

From: Eliot Lear <[email-masked]>
Date: Fri, 19 Sep 2014 10:42:57 +0200

<chair hat off>

Paul wrote:

> I suggest removing the last paragraph of section 4.2.3 for two
> reasons. First, there is a provision for handling deleted timezones,
> namely, retrieve the database without using "changedsince". Second, it
> is not bad practice to delete timezones. For example, tzdata's release
> 2014f deleted the zone Asia/Harbin?
> <http://trac.tools.ietf.org/wg/tzdist/trac/wiki/Asia/Harbin>
> (replacing it with a backward-compatibility alias) as its data turned
> out to have been largely imaginary.

This matter was partially addressed with the inclusion of aliases in
Section 3.2 of -01.  Therefore, I suggest rewording of the paragraph,
rather than its removal, as follows:

OLD:

   Note, this process makes no provision for handling deleted time
   zones.  In general it is bad practice to delete time zones as they
   might still be in use by consumers of time zone data.

NEW:

   Note, while this memo discuss the use of aliases, outright deletion of
   time zone names that may be in use by consumers is inadvisable, as
   their devices may contain events, both past and future that could no
   longer be resolved into a set time.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Fri, 19 Sep 2014 10:52:14 +0100

>    longer be resolved into a set time.

This may also link in with the TZID overlap 'problem'.
Since my own target applications are for normalizing historic data then
I want a 'namespace' that is consistent, so there would be no provision
for deleting TZID's that may already exist in the normalized data! I
still see problems with "changedsince" where tz's '1970 normalizing'
moves data from one alias to another.

This next comment may be more appropriate against ticket 9 ...
I'm still not sure that we have a workable answer to the problem of
retrieving a snapshot of the the tz data at a particular point in time?
It is not practical to rebuild archival data every time a TZID update
happens, so identifying the base namespace and version so that new users
can access a correct view is an important part of any tzdist system.
Simply assuming that a 'current' view is all that is needed is somewhat
restrictive, and if different namespaces are using different names then
many time zones may appear as deleted anyway?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: Eliot Lear <[email-masked]>
Date: Fri, 19 Sep 2014 12:05:04 +0200

> moves data from one alias to another.

I think you're right to address this through the TZID overlap problem. 
That also makes clear that management of TZIDs becomes a matter for each
publisher, who is responsible for consistency.

Eliot

---

From: Tobias Conradi <[email-masked]>
Date: Fri, 19 Sep 2014 13:48:07 +0200

>    longer be resolved into a set time.
What is the difference between "outright deletion" and "deletion"?

What is "set time"?

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Lester Caine <[email-masked]>
Date: Fri, 19 Sep 2014 13:02:19 +0100

> What is the difference between "outright deletion" and "deletion"?

Removal of an item rather than simply identifying it as deleted.
When reviewing historic material it can often be important to identify
transient material and so delete should always be a matter of simple
tagging it's existance and reasons for deleting rather than "outright
deletion". Since tzdist has no involvement in the actual material, all
we can control is the TZID management, but even that is not a charter
requirement?

> What is "set time"?
Does require clarification, but essentially if a time is based on
material that no longer exists it can't be used to set a physical time.

---

From: Eliot Lear <[email-masked]>
Date: Fri, 19 Sep 2014 14:10:08 +0200

Hi Tobias,
> What is the difference between "outright deletion" and "deletion"?

The term "outright" is meant as emphasis with negative connotation.

> What is "set time"?

Yes, that's a fair question.  How about instead "a time that can be
unambiguously determined by the client"?

Eliot

---

From: Arthur David Olson <[email-masked]>
Date: Fri, 19 Sep 2014 09:11:31 -0400

> How about instead "a time that can be unambiguously determined by the
client"?

To reduce overloading of "time," change "a time" to "an instant"?
Here and elsewhere?

    --ado

On Fri, Sep 19, 2014 at 8:10 AM, Eliot Lear <[email-masked]> wrote:

> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist
>
>

---

From: Tobias Conradi <[email-masked]>
Date: Sun, 21 Sep 2014 10:09:32 +0200

> The term "outright" is meant as emphasis with negative connotation.
Is that needed in a technical document?

> unambiguously determined by the client"?
I don't know, I still try to understand the meaning of the text.

E.g. "time zone name" is not listed in the glossary.

Also "past and future" excludes current.

And "in use by consumers" does that always mean "their devices may
contain events"? If yes, it is a duplication, if no, then there are
uses, that are excluded somehow from the reasoning by the part that
talks about devices.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Paul Eggert <[email-masked]>
Date: Sun, 21 Sep 2014 13:50:49 -0700

>     longer be resolved into a set time.

I'm not sure it's "inadvisable".  As I understand it, FreeBSD has been 
doing it for many years, and even the tz database has done it on 
occasion, so on what bases would we declare this practice "inadvisable"? 
  If it's known that a time zone name is rarely used, or is typically 
used incorrectly, it's not necessarily inadvisable to withdraw it.

Clients typically substitute UTC for withdrawn or unknown time zone 
names (perhaps we should document this...) and often this is good enough 
-- in some cases it can be better than the behavior of the withdrawn name.

If we want to discuss this issue rather than remove the paragraph, I 
suggest that we document common practice, by replacing the last 
paragraph of 4.2.3 with two paragraphs, as follows:

OLD:

Note, this process makes no provision for handling deleted time zones. 
In general it is bad practice to delete time zones as they might still 
be in use by consumers of time zone data.

NEW:

If a time zone X is deleted, it is common for a publisher to make its 
identifier an alias for some other time zone Y, to cater to clients that 
may still be using X's identifier; because this changes Y, it can be 
discovered by a "list" with "changedsince".  In contrast, the outright 
deletion of X (with no other change) requires a "list" without 
"changedsince" to be discovered.

It is common for clients to silently substitute UTC for withdrawn or 
unknown time zone names.

---

From: Stephen Colebourne <[email-masked]>
Date: Sun, 21 Sep 2014 22:40:35 +0100

On 21 September 2014 21:50, Paul Eggert <[email-masked]> wrote:
> identifier an alias for some other time zone Y, to cater to clients that may
> still be using X's identifier; because this changes Y, it can be discovered
> by a "list" with "changedsince".  In contrast, the outright deletion of X
> (with no other change) requires a "list" without "changedsince" to be
> discovered.
>
> It is common for clients to silently substitute UTC for withdrawn or unknown
> time zone names.

The practice of substituting UTC for unknown zone was a major issue
with older versions of the JDK. In practice it meant that a zone that
is spelled incorrectly returned UTC rather than a meaningful error.
For the proposed text above to work as desired, it needs to be
possible to distinguish between a spelling error and a "removed" ID.

Personally, I have frequently called for more stability in the TZDB,
including in IDs. The concept of deleting IDs without aliases, is
highly undesirable.

Stephen

---

From: Eliot Lear <[email-masked]>
Date: Mon, 22 Sep 2014 07:42:56 +0200

Paul,

On 9/21/14, 10:50 PM, Paul Eggert wrote:

> identifier an alias for some other time zone Y, to cater to clients
> that may still be using X's identifier; because this changes Y, it can
> be discovered by a "list" with "changedsince".  In contrast, the
> outright deletion of X (with no other change) requires a "list"
> without "changedsince" to be discovered.
>

I'm ok with that text.  It warns publishers of potential consequences
without getting too far into their business.

Eliot

---

From: Tobias Conradi <[email-masked]>
Date: Mon, 22 Sep 2014 12:17:13 +0200

On Sun, Sep 21, 2014 at 10:50 PM, Paul Eggert <[email-masked]> wrote:
> I'm not sure it's "inadvisable".
I am sure it breaks things.

> As I understand it, FreeBSD has been doing
> it for many years,
So they broke things for many years. Thanks for telling.

> and even the tz database has done it on occasion,
No surprise.

> so on
> what bases would we declare this practice "inadvisable"?
No idea. I cannot see that the charter of tzdist allows for handing
out advices to database maintainers.

>  If it's known that
> a time zone name is rarely used,
Rare breaks are breaks too.

> or is typically used incorrectly,
How that?

> it's not
> necessarily inadvisable to withdraw it.
Advices vary.

> Clients typically substitute UTC for withdrawn or unknown time zone names
> (perhaps we should document this...)
How? If data is stored as 15:00 time zone identifier: <time zone
identifier>, how can there be made any claim regarding UTC?

I skip the rest of the email.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Lester Caine <[email-masked]>
Date: Mon, 22 Sep 2014 13:07:56 +0100

> It is common for clients to silently substitute UTC for withdrawn or

This is not relevant to tzdist?

One could argue that since the actual data is outside the scope of the
charter what is valid data is also irrelevant but I think we have to
agree that tzdist should provide a mechanism to at least identify
transient changes? That a publisher has provided a set of TZIDs makes
them responsible to maintain that list? That at least one source will
always maintain that list without any delete mechanism is also a given.

BUT
Once a list has been published, all of the entries will need to be
assumed to be used, so what the tzdist protocol needs to provide is the
mechanisms to manage changes to that list?

Case 1. timezone is deleted completely ... rather than then returning
'invalid TZ' we should return 'deleted ID' so that the client
application knows how to handle the situation. Giving UTC is not an option!

Case 2. timezone is merged ... This one should just work since the
mechanism will simply return a changed set of data. The reason for
flagging it is that while the "changed since" data may NOT have changed,
I would prefer that the change of 'status' would be flagged? One for
flagging when a record is tagged as 'changed', but I think we then need
some flag to show what the change is so again the client application can
be advised of the change.

Case 3. New timezone added ... Just including this for completeness!
Except.
I'm a little unsure just how this IS handled. If we look for a "changed
since" on the list, then any new TZID's will be listed, any changes to
aliases will update the relevant master TZID, but there is no indication
if a TZID has been removed. I think I would like to see a listing with a
"deleted" flag so that one CAN then review material which currently use
the TZID rather than waiting for it to fail later?

Of cause 'preferred practice' would never delete anything and a data
model NOT based on VTIMEZONE would simply use generic rules to build
TZID data in place of aliases per-say.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 22 Sep 2014 08:59:22 -0700

Lester Caine wrote:
> what the tzdist protocol needs to provide is the

There is such a protocol; namely, use "list", retrieve all the data, and 
inspect it for changes.  The entire database can be communicated in less 
than 32 KiB with a reasonably-compressed format so this should not be 
that big a deal for a secondary server.

The topic came up only because the draft says one can retrieve changes 
more efficiently by using "list" with "changedsince", an optimization 
that does not work for deleted zones.

By the way, given the small amounts of data involved, it's not clear 
that "changedsince" is a win; perhaps we should remove "changedsince" in 
the interest of simplicity.

> rather than then returning 'invalid TZ' we should return 'deleted ID'

Let's not complicate the protocol to attack this relatively minor 
problem.  If we wanted to head down this direction, we should let 
clients query the database history.  But that sort of thing is overkill 
for this iteration.

---

From: Lester Caine <[email-masked]>
Date: Mon, 22 Sep 2014 17:30:43 +0100

> the interest of simplicity.
That perhaps less than half a dozen TZID's will change and ONLY material
relating to those may need review, getting every client to reprocess the
whole list seems major overkill.
If a simple internet device is ONLY using the deleted TZID, then what
happens when it asks to list just that ID?

> clients query the database history.  But that sort of thing is overkill

I beg to differ. Simply ignoring the problems created by not taking care
of historic uses of material will be another black hole later. It is the
very problem OF querying database history that I would prefer at least
to be protected initially. My preferred fix for this particular aspect
is simply to prevent any TZID from being deleted so I don't have a
problem what ever is done 'this iteration', but if a publisher does
start deleting TZID's from data currently in use and the protocol allows
it, it should also properly flag that it has done.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 22 Sep 2014 12:44:55 -0400

Hi Paul,

--On September 22, 2014 at 8:59:22 AM -0700 Paul Eggert 
<[email-masked]> wrote:

> By the way, given the small amounts of data involved, it's not clear that
> "changedsince" is a win; perhaps we should remove "changedsince" in the
> interest of simplicity.

Whilst the amount of data one client receives is small, the amount the 
server has to serve up maybe huge depending on the number of clients. For a 
large service provider or an OS vendor wanting to use this protocol, that 
can map to significant cost increases. We had the same issue with CalDAV 
when we originally started deploying it - it lacked a "sync" REPORT and 
clients would do PROPFIND Depth:1 requests (equivalent of a full listing of 
a directory) to sync calendars. This proved to be costly when scaling to 
10's of millions of users or more. Adding the sync REPORT made a big 
difference.

So my preference is to stick with changed since. It is relatively easy to 
implement server side and I believe will make a big difference in 
scaleability.

> problem.  If we wanted to head down this direction, we should let clients
> query the database history.  But that sort of thing is overkill for this
> iteration.

That said, I don't think it would be hard for the server to track deletions 
and report them in a changedsince response. But perhaps a better approach 
for the server is to "freeze" the time zone data at the last state it had 
prior to deletion and add a TZUNTIL to indicate that it is no longer valid. 
In effect that is overriding the publisher's "delete" action by making the 
time zone as invalid past the deletion point. That at least allows clients 
that still have references to the tzid to continue to use it for past data 
- though that may not match the actual intent of the removal.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Mon, 22 Sep 2014 09:53:50 -0700

> happens when it asks to list just that ID?

It gets the same error response it would get when listing any invalid 
ID, which is good enough to address this problem.

---

From: Tobias Conradi <[email-masked]>
Date: Mon, 22 Sep 2014 19:01:05 +0200

On Mon, Sep 22, 2014 at 5:59 PM, Paul Eggert <[email-masked]> wrote:
> than 32 KiB with a reasonably-compressed format [...]
Which database? I thought the TZDIST service is meant to work for
different databases?

A TZDB could be easily ten times the size of any currently existing TZDB.

> The topic came up only because the draft says one can retrieve changes more
> efficiently by using "list" with "changedsince", an optimization that does
> not work for deleted zones.
Why not? Deletions could be logged by the time zone server.

>> rather than then returning 'invalid TZ' we should return 'deleted ID'
ID is more in line with the glossary, even better might be TZID since
the glossary lists "Time Zone Identifier".

So it would be 'invalid TZID' and 'deleted TZID'.

-- 
Tobias Conradi
Rheinsberger Str. 18
10115 Berlin
Germany

http://tobiasconradi.com

---

From: Lester Caine <[email-masked]>
Date: Mon, 22 Sep 2014 18:14:30 +0100

On 22/09/14 17:44, Cyrus Daboo wrote:
> That said, I don't think it would be hard for the server to track
> deletions and report them in a changedsince response. But perhaps a
> better approach for the server is to "freeze" the time zone data at the
> last state it had prior to deletion and add a TZUNTIL to indicate that
> it is no longer valid. In effect that is overriding the publisher's
> "delete" action by making the time zone as invalid past the deletion
> point. That at least allows clients that still have references to the
> tzid to continue to use it for past data - though that may not match the
> actual intent of the removal.

This is another approach. The example I was working through is a 'simple
internet clock' and in the case of the used TZID being deleted what is
the better fallback state while ALSO displaying an error. In the case of
'invalid ID' you only have the possibility of switching the display to
UTC, while 'deleted ID' allows the display to remain using the last offset.

A responsible publisher would certainly always retaining used TZID's but
would also supply information on what a TZID has been switched to. That
may depend on new location data so is certainly outside the scope, but
once again the lack of a reason mechanism within the protocol prevents
supplying a reason.

---

From: Paul Eggert <[email-masked]>
Date: Mon, 22 Sep 2014 10:34:29 -0700

On 09/22/2014 09:44 AM, Cyrus Daboo wrote:
> my preference is to stick with changed since

Fine with me (I was just trying to make your life easier :-).

> perhaps a better approach for the server is to "freeze" the time zone 
> data at the last state it had prior to deletion and add a TZUNTIL to 
> indicate that it is no longer valid.

I'm not sure that would be better.  Often a zone is withdrawn not 
because its data are wrong from the withdrawal date on, but because its 
data are wrong for previous (historical) dates.  We should try to keep 
metadata (when did we change the database?) separate from data (what 
temporal periods do the database's time zones cover?).

---

From: Tim Parenti <[email-masked]>
Date: Wed, 24 Sep 2014 11:29:01 -0400

> without getting too far into their business.

Bringing this discussion back to changes to the text of the draft, I offer
my +1 to the above and echo Eliot's reasoning.

--
Tim Parenti

---

From: Daniel Migault <[email-masked]>
Date: Wed, 24 Sep 2014 18:37:45 +0200

Hi,

This thread has had quite a lot of cross discussions. It seems that most
people agree with the following text:

"If a time zone X is deleted, it is common for a publisher to make its
identifier an alias for some other time zone Y, to cater to clients that
may still be using X's identifier; because this changes Y, it can be
discovered by a "list" with "changedsince".  In contrast, the outright
deletion of X (with no other change) requires a "list" without
"changedsince" to be discovered."

It is not clear to me whether the following text is a recommendation or a
statement.
"It is common for clients to silently substitute UTC for withdrawn or
unknown time zone names."

I am wondering if we should add some text whether such practices are
recommended or not, and if an error or a specific time zone
(non-existing-time-zone) shoudl be used.

BR,
Daniel

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Paul Eggert <[email-masked]>
Date: Wed, 24 Sep 2014 12:15:57 -0700

On 09/24/2014 09:37 AM, Daniel Migault wrote:
>
> It is not clear to me whether the following text is a recommendation 
> or a statement.

When I wrote it I thought of it as a statement, i.e., information that 
could be of use to authors of client or server software.  It'd be OK to 
turn it into a recommendation.

> (non-existing-time-zone) shoudl be used.

tzdist should not recommend an error, or any specific time zone other 
than UTC.  That would disagree with the common practice of silently 
substituting UTC.  There's no reason to disallow this common practice.

---

From: Daniel Migault <[email-masked]>
Date: Wed, 24 Sep 2014 21:27:51 +0200

Thanks Paul for the clarification.

On Wed, Sep 24, 2014 at 9:15 PM, Paul Eggert <[email-masked]> wrote:

> tzdist should not recommend an error, or any specific time zone other than
> UTC.  That would disagree with the common practice of silently substituting
> UTC.  There's no reason to disallow this common practice.
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 24 Sep 2014 15:37:25 -0400

Hi Paul,

> substituting UTC.  There's no reason to disallow this common practice.

There is another possibility which is to treat the time as "floating" - 
something that iCalendar allows. i.e., treat it as a local time in whatever 
time zone the client is using for showing events to the user.

I do agree that we shouldn't recommend a particular client behavior, but 
the question of describing common behavior may not be clear-cut.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Wed, 24 Sep 2014 22:38:11 +0100

On 24/09/14 20:37, Cyrus Daboo wrote:
> something that iCalendar allows. i.e., treat it as a local time in
> whatever time zone the client is using for showing events to the user.
> 

Common practice at one time was to only use two digits for the year. In
some areas we are still dealing with the fallout!

In the context of tzdist ... if a TZID has been published by a server,
then we could simply require that it is maintained even if it is just
transferred to an alias on another one. Allowing a publisher to simply
drop a TZID that is currently being used by clients requires the client
to take some action, and 'treating it as a local time' requires the
client maintains the last offset rather than switching the clock to UTC.

Currently there is no reliable 'service' to provide TZ data but once it
is available then like time servers it will become relied on and what is
currently acceptable practice may no longer be appropriate? For simple
clock based systems, loading them down with managing the availability of
a TZID table to check what are valid is the problem here. That an
operating system can handle TZ changing and dropping TZID's is one
thing, but for simple devices a simple solution should be supported, and
if the TZID they are using disappears I think I would ACTUALLY say this
is not acceptable, but rather than simply now returning 'invalid' I feel
the server SHOULD return a valid replacement somehow so that the device
can continue working? If it has to switch to UTC then I would also
recommend that the device would simply switch off for safety reasons!

It is a problem for the publisher, but what we serve to the client must
remain consistent once we are providing the service, and the protocol
needs to handle every scenario in a consistent way.