Subject: Re: [Tzdist] [tzdist] #32 (service): managing historical data

From: Daniel Migault <[email-masked]>
Date: Sat, 1 Nov 2014 14:14:31 +0100

Hi Lester,

Given the new URL format discussed in Issue #29. A version has been
specified. Can you let us know if it solves the historical data issue
completely. If it does not solve it completely, could you expose the
appropriated extensions that would solve this issue?

>From the call conference, I thought that this issue would be solved by
introducing redirections to tzids. That is if I need the complete time zone
related to Paris from let say 1875 to 2056, I would be returned a list:
 [{utc_start:start_time0,
   utc_stop:stop_time0,
   tzid: tzid0}
   ....
   {utc_start:start_timeN,
   utc_stop:stop_timeN,
   tzid: tzidN}
]

where tzid is one of the Time zones we are distributing. I might be
completely wrong. Suppose i am not, I am not also sure that such a
mechanisms will be described in the current service distribution. It might
be another document. However, the current service distribution should
clearly make that possible.

BR,
Daniel

On Sat, Oct 25, 2014 at 1:24 AM, Lester Caine <[email-masked]> wrote:

> On 24/10/14 20:05, Tobias Conradi wrote:
> >> You are right.
> > No he is not. If the "need to put local time into a log for some
> > reason", then it does not mean that "they can log UTC offset".
>
> Log sequence is gauranteed by the use of UTC base, and since we do not
> need to know about any other time than the logged time, the offset is
> all that is needed. BUT the audit trail still needs to know if the
> correct offset was used for that offset which is where a version stamp
> is important.
>
> --
> Lester Caine - G8HFL
> -----------------------------
> Contact - http://lsces.co.uk/wiki/?page=contact
> L.S.Caine Electronic Services - http://lsces.co.uk
> EnquirySolve - http://enquirysolve.com/
> Model Engineers Digital Workshop - http://medw.co.uk
> Rainbow Digital Media - http://rainbowdigitalmedia.co.uk
>
> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Lester Caine <[email-masked]>
Date: Sat, 01 Nov 2014 15:37:33 +0000

> appropriated extensions that would solve this issue?

Introducing a correctly managed version number in the first instance
fixes the race problem created by the current 4.2.2 and 4.2.3 !
A set of published calendar data will somehow identify the version of
tzdist data it is currently using, and a new client can request that tz
data set at the same time as accessing the calendar data. If the tzdist
source has been updated ( say overnight ) but the calendar data has yet
to be revised, then the client can identify potential problems for
themselves while the current process WOULD return a later tzdist set
than that actually being used in the calendar.

Historic material just gets automatically tidied up in the base fix at
the expense of maintaining a growing set of versions over time. As has
been said, the process needs to work from day one and is not something
that can be added in 10 years time :(

> introducing redirections to tzids. That is if I need the complete time
> zone related to Paris from let say 1875 to 2056, I would be returned a list:
>  [{utc_start:start_time0,
>    utc_stop:stop_time0,
>    tzid: tzid0}
>    ....
>    {utc_start:start_timeN,
>    utc_stop:stop_timeN,
>    tzid: tzidN}
> 
> mechanisms will be described in the current service distribution. It
> might be another document. However, the current service distribution
> should clearly make that possible.

THAT summarises where I think I've got to myself quite nicely! expand
can return usable data across essentially 'rule' boundaries, but I'm not
sure if iCalendar can handle a switch of rule sets perhaps 'mid rule' so
we get say two summertime offsets following one another rather than a
clean summertime/wintertime sequence?  I currently envisage tzidN as
perhaps a current generic rule set, while tzid0 may simply be a location
identifier rather than defining a full set of rules. THAT is why I'm
thinking the prettyURL (That is what RESTFUL style urls WAS called)
field IS 'rule', and this composite set becomes a 'zone' or 'location' ...

---

From: Lester Caine <[email-masked]>
Date: Fri, 28 Nov 2014 14:14:12 +0000

Copy of thread with Daniel Migault off list ... with a couple of
spelling corrections ;)

On 28/11/14 10:06, Daniel Migault wrote:
> Hi Lester,
> 
> So just to check I understand properly. From the conference call, I
> though that the issue was that there were no mapping from one place to
> tzid, and the fact that tzid embeds locations names may be misleading.
> The case i had in mine was that city may be using tzid1 from d0 to d1,
> tzdi2 from d1 to d2 and so on... Just to make it clear this is not the
> case we are discussing now and this mapping is not in scope of the draft.

The fact that tzdist has no control over the ID's used is a problem, but
it's not one I am discussing here. If I publish a set of event related
data be it current calendars or historic material then I ALSO have to
include the 'publisher' so that I can gain access to the correct set of
ID's along with the 'version' so that I get the matching set of data to
go with those ID's ... even for current material there is no way to
avoid the 'version' since that may change over night on tzdist, but the
material using it may be cached somewhere and still require the previous
version.

> My understanding of your email is that I am publishing an event at d0,
> this event happens at d1. When I am publishing the event, the tzid
> considered was tzid0.
> Suddenly an change occurs in tzid0, there are the situation I envision:
>     - Change occurs before d0: no problem
>     - Change occurs after d1: no problem
>     - Change occurs between d0(UTC) and d1(UTC), we have an issue
> because the d1(local) initially foreseen at d0 may be different from the
> one that actually happen. Is that the issue you raised?

In a small number of cases when a transition from d0 to d1 happens is
determined by some astronomic observation. My own experiences of this
relate to the Middle East and there have been events messed up because
the local clocks changed and the on-line material did not. NOWADAYS it's
practice to just avoid these periods when planning meetings, but the
system needs to be able to at least pick up that there may be a problem
rather than simply publishing unreliable data? If the calendars version
is X and tzdist is providing X+1 one at least has a chance of checking
if the events you are using could be affected by the '+1' change rather
than waiting for the source data to be updated.

> I do not have a great experience in calendaring so I might be completely
> out of scope, and feel free to let me know when I am completely wrong. I
> would think that we do not need version of the tzid as long as the event
> carries the creation date.

This is more important on historic material. We may create a set of
material normalized on a particular 'creation date', but we still need
to be able to gain access to the version of tzdist data used at that
time. With this process inluded, the 'overnight' changes are also
cleanly identified.

> The other think I do not see clearly is what is he benefit of versioning
> versus changing the name of the tzid.

Different 'publishers' can have different sets of ID's and the name
defines that, but even if the 'version' is added as part of that name,we
still need the versioning process to ensure everybody gets the correct
set of data today or in any time in the future.

> On Wed, Nov 26, 2014 at 12:45 PM, Lester Caine <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On 26/11/14 09:11, Daniel Migault wrote:
>     > It seems that the WG is converging over a URL format, so I believe it
>     > may be time to address Issue #32
>     > <http://trac.tools.ietf.org/wg/tzdist/trac/ticket/32> managing
>     > historical data.
>     >
>     > Could you elaborate given the URL format described in draft v3 a
>     > mechanism that makes possible historical data management and respond to
>     > this mail
>     > <http://www.ietf.org/mail-archive/web/tzdist/current/msg00905.html>?
>     >
>     > The primary goal is to check that the current URL format makes this
>     > possible. Then we will see if the mechanism should be included in the
>     > current draft or as an additional draft.
> 
>     Basically managing historic data simply falls out naturally when you
>     include the fix for the race hazard on synchronising CURRENT data.
> 
>     It is simply /distribution/xxxx/version/yyyy/
> 
>     Even with current material, a calendar data set HAS to specify which set
>     of data they are using, which is either the base URL of the server, or
>     the xxxx when accessing a more general distribution path. On top of this
>     the correct version needs to be identified as the current 4.2.2 CAN NOT
>     be relied on to ensure that the same version of tzdist data is picked up
>     by a new user.
> 
>     If I publish an event diary today and say it's version 'today' of the tz
>     data, and the tz data gets a short term correction tonight rendering the
>     published data incorrect, then I either need to update the data OR a new
>     user trying to access the data tomorrow will see *I* am using todays
>     data but the tzdist is giving him a later version, so he needs to ACCESS
>     the old version to see if the data he is looking at is still correct.
> 
>     Once the version of published data is released that becomes static
>     information, and in N years time I can look at the same event data and
>     access the correct version of tz data to read it. In the case of a
>     current calendar it will be appropriate to update the information and
>     release it with a current set of tz data, identifying the new tz
>     version, but archived data only requires the original data set. There
>     should be no need to reprocess that data unless it's identified that the
>     original data set was incorrect.
> 
>     I'm sure you can see that as has been said before, if we are using
>     tzdist data today we also need to identify and maintain each new
>     release. This is not something that can be fixed in 10 years time, but
>     it also ensures that the correct current data is being used!

---

From: Daniel Migault <[email-masked]>
Date: Sat, 29 Nov 2014 14:55:23 +0100

So I am trying to summarize the use we are addressing. Suppose we are
planning an event at date D0 using UTC as a reference, for people in
different time zone. This event is planned at date D1. For one of the
participant, its time zone has been updated between D0 and D1. This updates
results in having D1 (UTC) to move from D1_LOCAL_0 before the change occurs
to D1_LOCAL_1 after the change.

If one receives the event indicating that D1 is D1_LOCAL_0. I should be
able to say, this local time has been derived from an tz that is not
relevant at D1, so so the actual event is at D1. Is that the or at least a
use case we intend to address?

If so, using version could indicate which 'tz' has been used. For my
information how do we specify, the reference time? In our example, I chose
UTC, but, if we had chosen local time of the changing time zone, then D1
would have been changed for almost all other time zones.

If the mentioned use case is correct, then we may have to look at:
    1) whether we have consensus that a version should be added
    2) How version can be inserted in the URL model
    3) What are the version speciifc operation. Maybe we should be able to
ask for the lastest version?

BR,
Daniel

On Fri, Nov 28, 2014 at 3:14 PM, Lester Caine <[email-masked]> wrote:

> >     > <http://www.ietf.org/mail-archive/web/tzdist/current/msg00905.html
> >?
> >     >

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Lester Caine <[email-masked]>
Date: Sat, 29 Nov 2014 15:20:10 +0000

On 29/11/14 13:55, Daniel Migault wrote:
> participant, its time zone has been updated between D0 and D1. This
> updates results in having D1 (UTC) to move from D1_LOCAL_0 before the
> change occurs to D1_LOCAL_1 after the change.
> 
> relevant at D1, so so the actual event is at D1. Is that the or at least
> a use case we intend to address?

I need someone more 'proficient' with Islamic Time to fill in some of
the fine detail, but the problem arises when a meeting scheduled for say
10AM local time in a day or so's time is published with a UTC time of X
but then there is a revision to the local time due to the observed
events which may start or stop Daylight Saving differently to expected
so the local time moves and hour relative to UTC. That a video linkup
will now happen differently was the problem I have had in practice, and
the local time of the event SHOULD have been changed since the link was
booked on the UTC time! Once you know the problem has arisen then there
are several ways to resolve it but all involve change one time or another.

The same problem will arise at short notice if a published DST
transition does not ACTUALLY happen. We have had a couple of examples of
this recently, and these might affect events longer into the future!

> information how do we specify, the reference time? In our example, I
> chose UTC, but, if we had chosen local time of the changing time zone,
> then D1 would have been changed for almost all other time zones.

The event is published using the current TZ data, and we have to publish
the version information with the event data. When a delegate looks at
the announcement for the event the local time advertised will be using a
particular publishers set of TZID's and the current version of that
data. By the time the delegate is looking at the event timetable, the
latest TZ data may already be 'current+1' as so the data presented may
be wrong. He then needs to know that for his particular TZID there is a
discrepancy between the data he is looking at and the CURRENT situation
on the ground. HOW the problem is to be fixed will depend on just what
can be moved ... the local time activity retaining the UTC time, or does
the UTC time need to change to reflect the new local time. *WE* can't
answer that via tzdist, but by proper use of the version we can identify
that there are two possible times now for this event and that the
delegate needs to perhaps update their data for the event to match the
LATEST version of the tz data or if no later version of event data is
available - contact the organizers.

>     1) whether we have consensus that a version should be added
I do not see how you can avoid a race condition without it?

>     2) How version can be inserted in the URL model
In order TO avoid the race condition a user must be able to access the
data that matches that used to publish some event. Secondary to that
this fixes all of the historic material since as new versions are
published, all of the historic data is retained, and there may be
several versions of tz between publishing an event calendar to the event
happening ... changedsince can not be used to ascertain if any of the
material has changed ... you either check out an updated version of the
event calendar with a later version of tz - if available, or
alternatively one can check if there is a difference between the version
used to publish the data and the current data.

>     3) What are the version speciifc operation. Maybe we should be able
> to ask for the lastest version?
What happens when the latest version has a short notice change in DST
data which is not yet handled in the material one is looking at? In
particular, LOCAL delegates may well be aware that there is some
uncertainty when an event will happen, but the people using UTC time are
generally totally unaware and in the current state of flux on many DST
transitions, one does need to know that there has been a change rather
than simply assuming that the 'latest' version is being used for the
data we are looking at! The local time of an event may well change if a
global link up has been booked based on UTC time.