Subject: [Tzdist] URL design

From: Stephen Colebourne <[email-masked]>
Date: Mon, 27 Oct 2014 21:01:39 +0000

The following is a refined proposal for the URL design of tzdist. It
is based on fairly well-known and widely used best practices, with one
discussion of the principles at this link:
http://codeplanet.io/principles-good-restful-api-design/
Also see this maturity model for RESTful design
http://martinfowler.com/articles/richardsonMaturityModel.html

a) HTTP verbs
HTTP provides distinct verbs for specific types of request, such as
GET, POST, DELETE and PUT. tzdist is a read-only specification, thus
only GET applies.

b) Primary resource concepts
The primary resource concept of tzdist is the time zone. The comes in
two flavours - a set of rules (VTIMEZONE format) and a set of
transitions (actual clock changes, observances or expanded format).

Other key concepts are lists of time zones, publishers and versions of
the data produced over time by publishers.

The distinction between a time zone identifier and a time zone alias
is marginal from the perspective of a consumer of the web service,
thus they are not treated as distinct resources.

c) Mapping to URLs
Each primary resource concept is mapped to a base URL, with query
strings used for filtering. Specifically, any URL may be requested
without a query string filter, in which case all applicable data will
be returned.

The following maps most closely to the current section 6:

6.1 Capabilities
Current:
GET /?action=capabilities

Proposed:
GET /capabilities

No query string.
Response changed to include HATEOAS URLs to locate the lists of rules
and transitions.

6.3 Get
Current:
GET /?action=get&tzid=America/New_York
      &format=text/calendar
      &truncate=2010-01-01T00:00:00Z,2019-12-31T11:11:59Z
Proposed:
GET /rules/America/New_York
      &start=2010-01-01T00:00:00Z
      &end=2020-01-01T00:00:00Z
Accept: text/calendar

Returns details of a single identifier in the form of VTIMEZONE type rules.
Header used for the MIME type.
Query parameter "start", optional, defaults to far past.
Query parameter "end", optional, defaults to far future.

6.4 Expand
Current:
GET /?action=expand&tzid=America/New_York
      &start=2008-01-01T00:00:00Z
      &end=2009-01-01T00:00:00Z

Proposed:
GET /transitions/America/New_York
     ?start=2008-01-01T00:00:00Z
      &end=2009-01-01T00:00:00Z

Returns details of a single identifier in the form of expanded observances.
Query parameter "start", optional, defaults to far past.
Query parameter "end", optional, defaults to far future.
Query parameter "changedsince" is probably not needed with standard
ETag caching.

6.2 List
Current:
GET /?action=list&lang=en_US

Proposed:
GET /rules
Content-Language: en-us

GET /transitions
Content-Language: en-us

Lists all identifiers, with optional local language.
The list operation for rules is separate for that of transitions.
This allows a server to have a different set of supported identifiers
for rules and transitions (although this is unlikely in practice).

Response changed to include HATEOAS URL for each identifier returned.
(It is reasonable to question whether the local names should be part
of this response, or only part of the more detailed single identifier
response)
Header used for the language.
Query parameter "changedsince" is probably not needed with standard
ETag caching.
Query parameter "pattern", optional, allows pattern match searches:

6.5 Find
Current:
GET /?action=find&pattern=US/Eastern

Proposed:
GET /rules?pattern=US/Eastern

GET /transitions?pattern=US/Eastern

d) Extensions
The URL design permits the spec to be extended from its current scope:

d.1) Publishers
The ability to clearly identify a publisher may be useful.

Proposed:
GET /publishers/{publisher}/rules

GET /publishers/{publisher}/transitions

Where {publisher} "iana" would be reserved.

d.2) Versions
The ability to clearly identify the version of the data may be useful.

Proposed:
GET /publishers/{publisher}/versions/{version}/rules

GET /publishers/{publisher}/versions/{version}/transitions

Where {version} "latest" would be reserved for the latest version.
Note that with this approach, the data at the following URLs must
never change and could be cached indefinitely:

GET /publishers/iana/versions/2014d/rules
GET /publishers/iana/versions/2014d/transitions

e) Spec changes
Section 5, discussion of request parameters and section 6 actions will
need rework.
Section 10 will also need rework, as actions are no longer present.

f) Comments
The main problem with the URL design is the fact that common time zone
identifiers have a slash in them. While the slashes could be encoded
in the URL, it seems preferable to avoid the encoding, which can be
achieved by putting the identifier at the end of the URL. This results
in two main options:

GET /rules/America/New_York
GET /transitions/America/New_York

GET /tzids/transitions/America/New_York
GET /tzids/rules/America/New_York

After consideration, the former is more powerful and no more complex,
thus is recommended.

thanks
Stephen

---

From: Sanjeev Gupta <[email-masked]>
Date: Tue, 28 Oct 2014 12:38:40 +0800

>       &end=2020-01-01T00:00:00Z

If the Accept: header is not sent, what would the response look like?

Thank you,
-- 
Sanjeev Gupta
+65 98551208     http://www.linkedin.com/in/ghane

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 28 Oct 2014 07:32:10 +0000

> If the Accept: header is not sent, what would the response look like?

"If no Accept header field is present, then it is assumed that the
client accepts all media types."
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Thus, the server gets to choose what format to send. In this case, the
spec should chose a format (JSON for consistency with the rest of the
spec)

Stephen

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 08:11:12 +0000

> http://codeplanet.io/principles-good-restful-api-design/

I assume that the intention here would be that the root URL can also be
used as part of the API. I think I am still envisioning timedb.co.uk/ as
the website with all of the documentation relating to the data sources,
while tz.timedb.co.uk/ accesses data and loc.timedb.co.uk/ will provide
a lookup service against a location. However a different sub-domain
would be used to provide - for example - a simple single set of data for
a clock service on a local network, and bypass using a TZID at all. I'd
probably expand the tz. in the case of providing different views of the
data rather than going down the /publishers/versions/ route as this may
be the same underlying source, but just a different truncation. tzfull.
has everything while tz2010. provides a more compact calender service.
Same base data - different visibility!

> transitions (actual clock changes, observances or expanded format).

I'm currently thinking that a slightly modified format to the expand
view is ACTUALLY all that is needed for handling the historic data.
Expand but with the start and stop dates for each of the base rule sets.
Something that would be returned by a loc. lookup on a place name ...

> thus they are not treated as distinct resources.
I'm getting more concerned that this will become a mess? It IS the rule
over location problem, and just what rule is used depends what a
location lookup returns. Out of scope anyway, but for example if I am
using /Europe/Isle_of_Man currently it is aliased to /Europe/London but
as already indicated there is no information on when the two sets of
rules merged, so should I even serve /Europe/Isle_of_Man at all in this
case? I have to rely on another service to provide the missing detail.

>       &start=2010-01-01T00:00:00Z
?start=2010-01-01T00:00:00Z ;)

>       &end=2020-01-01T00:00:00Z
How would a different format be flagged? (Think you just answered that ... )

>       &end=2009-01-01T00:00:00Z
Omitting TZID and returning a default value works here? There is no need
to produce a list of TZID's? While /rules/ would produce a list of TZID's?

> Query parameter "changedsince" is probably not needed with standard
Does ETag actually take on board the filter values? I thought that was
one of the problems with using it here. It is establishing if the next
published transition has changed which is all that matters in the simple
case, and I'm not sure this works?

> Where {publisher} "iana" would be reserved.

PLEASE can we reserve 'iana' and 'iana1970'. I know that we can't make
any rules here, but since there are two distinct underlying views of the
data, starting from a base of the zone1970.tab indicates that the
published data IS an already truncated view.

> GET /publishers/iana/versions/2014d/transitions
What doe these return when on TZID is applied?

> achieved by putting the identifier at the end of the URL. This results
If one was starting again then things might be different. The '/' is a
pain in several places, such as creating pages in urls, so a preferred
method for encoding may be useful but not compulsory?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

---

From: "Clive D.W. Feather" <[email-masked]>
Date: Tue, 28 Oct 2014 09:09:29 +0000

Lester Caine said:
> PLEASE can we reserve 'iana' and 'iana1970'.

It would seem reasonable to reserve all strings beginning "iana".

-- 
Clive D.W. Feather          | If you lie to the compiler,
Email: [email-masked]     | it will get its revenge.
Web: http://www.davros.org  |   - Henry Spencer
Mobile: +44 7973 377646

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 28 Oct 2014 09:10:33 +0000

On 28 October 2014 08:11, Lester Caine <[email-masked]> wrote:
> used as part of the API.

The root URL can be anything. Other sections of the spec cover finding
the root URL.

> Something that would be returned by a loc. lookup on a place name ...

It is easy to imagine an extension to the proposed spec to support locations.

> case? I have to rely on another service to provide the missing detail.

It is important to remember that a spec like this should be written to
allow ease of consumption. Ease of server-writing is a secondary
concern.

The server-writer gets to determine whether Europe/Isle_of_Man and
Europe/London are the same, and they will be based on the standard
iana data. A different publisher or a separate location lookup service
handles the additional need of the different historical data (which
are far beyond what the web has today, and thus perhaps far beyond
what should be covered by this spec at all).

> case, and I'm not sure this works?

I didn't fully understand changedsince, so it is unclear how it maps
in. Caching gets you a long way I suspect.

> published data IS an already truncated view.

The term "iana" should refer to whatever the IANA group deems to be
its primary set of data. Other names could be reserved, or could be
freely up to the server-author.

GET /publishers
would list all available publishers.

> method for encoding may be useful but not compulsory?

URL encoding can already handle this, but it makes it harder for
tzdist consumers.

Stephen

---

From: Ken Murchison <[email-masked]>
Date: Tue, 28 Oct 2014 06:46:16 -0400

> in. Caching gets you a long way I suspect.
A client *could* do a conditional request for the list with an ETag, but 
that only tells the client that something in the list changed. If a 
client uses "changedsince" when requesting the list, the server will 
tell the client WHAT changed by only including those TZIDs that have 
changed since the dtstamp specified.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 10:51:45 +0000

> in. Caching gets you a long way I suspect.

It is the one sticking point which ever way one is working?
"changedsince" only works if you already have a cached copy of the data
from which there is a change. I still can't get my head around someone
subscribing to a new set of calendar based material and then accessing
the advised tz source to go with it. How does one know that the current
view of the tz data IS contemporary with that used to create the current
view of the calendar data? This is where I still feel that publishing
the current version being used is essential to keep everything in sync?
ETag similarly is of no use in these circumstances?
One can't simply assume that the user already has the relevant data from
which to use "changedsince"

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 28 Oct 2014 11:47:19 +0000

On 28 October 2014 10:46, Ken Murchison <[email-masked]> wrote:
> that only tells the client that something in the list changed. If a client
> uses "changedsince" when requesting the list, the server will tell the
> client WHAT changed by only including those TZIDs that have changed since
> the dtstamp specified.

I think that works fine as another query parameter on the relevant
URLs. In essence it is a request to return a filtered subset of the
total data available at that URL.

Stephen

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 28 Oct 2014 11:53:59 +0000

> which to use "changedsince"

I do think "changedsince" is hard for the server to implement. It
requires storing the instant in time that the data set of the server
was updated (which is unrelated to the version of the publisher and
unrelated to the instant that the publisher published the data). Given
that instant, the server has to check if the particular piece of data
has any changes between the requested instant and now. Note that the
server has to retain all data it ever served in order to be able to
implement this accurately (like a version control system).

It is certainly a lot easier for the server to just send down the
current entire set of data when a client determines that the ETag
cache key has changed.

Personally, I'm unconvinced that changedsince needs to exist.

Stephen

---

From: Ken Murchison <[email-masked]>
Date: Tue, 28 Oct 2014 07:56:06 -0400

On 10/28/2014 06:51 AM, Lester Caine wrote:
> from which there is a change.

Correct.  "changedsince" is only used iff the client has a previous 
"dtstamp" from the same server.  If the client doesn't have a "dtstamp" 
from the server, then it issues a request for the list without 
"changedsince" to fetch the entire list of TZIDs and the current "dtstamp".

> I still can't get my head around someone
> view of the calendar data?

The only data used by creator of an iCalendar event/task is a reference 
to a TZID.  It is up to the consumers of this iCalendar data to use a 
current VTIMEZONE corresponding to the referenced TZID.  There are 
recommendations in the draft on how a client/consumer keeps its data up 
to date.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 28 Oct 2014 13:02:22 +0100

> implement this accurately (like a version control system).

If it were a range in time, like show me everything that changed between
10 Apr 1992 00:00 GMT and 25 Oct 1996 00:00 GMT, then I would agree. 
But if we really mean changed since a time, I don't see why that can be
implemented as easily as performing an fstat() on a file for a compiled
zone, based on whether it in fact changed (that is, don't recompile
things that haven't changed).  If you want to avoid the fstat() then
cache the results.  But...
> Personally, I'm unconvinced that changedsince needs to exist.

If many others agree with you, then it doesn't matter how simple or
complicated it is to implement.

Eliot

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 12:28:48 +0000

> implement this accurately (like a version control system).

I had thought we had already agreed that the dtstamp would match the
creation date of the set of data being used, and that publishing an
update would not make any changes to a data set until the whole update
has been uploaded. A slave server will only transmit a stable snapshot
of the data and not allow elements to change in isolation. Which is
essentially why working with a simple version id is the only practical
approach. Until the next version is fully assimilated and any specialist
views are built, 'latest' will always return the previous version.
Essentially the ETag of GET /versions/latest/ will cleanly flag an
update is available, and a calendar using site can publish the version
currently being used by their data so that changes propagating through
the system ... which could potentially take days to fully ripple through
... do not affect published and quite incidentally archived material. A
new user accessing a calendar site or the first time can identify which
data set to use, and by use of /versions/latest/ can also see if any of
the data may be affected by a short term change that the calendar's
source has not yet got around to processing. No race conditions and
everybody knows what they are looking at!

---

From: Rob Seaman <[email-masked]>
Date: Tue, 28 Oct 2014 05:29:32 -0700

Temporal database researcher Richard Snodgrass is a collaborator on an observatory project to classify celestial transient events.  Snodgrass's concepts of valid time and transaction time are pertinent to this discussion:

	http://www.cs.arizona.edu/~rts/publications.html#temporalimplementation

That said, Eliot's observation seems correct.  Similar timestamps are used when data need to be detected as stale for instance statistics calculated for an astronomical image.  If the image file's modification time is more recent than the timestamp then the statistics have to be refreshed.

Rob Seaman
National Optical Astronomy Observatory
Tucson, AZ
--

> On Oct 28, 2014, at 5:02 AM, Eliot Lear <[email-masked]> wrote:
> 
> 
> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist

---

From: Ken Murchison <[email-masked]>
Date: Tue, 28 Oct 2014 08:37:13 -0400

> cache the results.  But...

We already have 3 servers that have it implemented quite easily and it 
works very well in keeping secondary servers and clients up to date with 
minimal traffic (e.g. not having to download the entire list for every 
update).

One can think of changedsince as a HTTP Last-Modified that works across 
multiple resources.

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 12:44:53 +0000

> The root URL can be anything. Other sections of the spec cover finding

4.2.1 Is only relevant in relation to discovering a service to use for a
deafult local source of data. SINCE there is no requirement that two
servers return the same data or even use the same set of ID's, a
consumer of tz data which is being used to build a calendar based
service could quite easily be using a different set of data and id's. So
I think the following text needs adding to 4.2.1 ...

" In addition to discovery of a default tzdist service, other users of
alternate services will need to provide access to the service they are
using to present their own material. This may not be based on the same
publisher as provided by the default service, so more than one service
may need to be utilised by a client. "

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 13:03:58 +0000

> minimal traffic (e.g. not having to download the entire list for every
Do you know how long it took the recent tz update to propagate through?

> One can think of changedsince as a HTTP Last-Modified that works across
I think 4.2.2 should perhaps be a little clearer that this IS how it
should work and that the 'returned timestamp' is the Last-Modified date
of the dataset. But it is still only practical if the 4.2.2 process is
not reading a newer version than that being used by the secondary data
server somewhere further down the chain ...

---

From: Ken Murchison <[email-masked]>
Date: Tue, 28 Oct 2014 09:17:45 -0400

> Do you know how long it took the recent tz update to propagate through?

No.  Our servers currently aren't syncing with each other.

---

From: Sanjeev Gupta <[email-masked]>
Date: Tue, 28 Oct 2014 21:56:31 +0800

> spec should chose a format (JSON for consistency with the rest of the

Thank you, that would be good.  I am looking at simple test scripts, where
the client may not support HTTP Headers.

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 28 Oct 2014 10:16:59 -0400

Hi,

> minimal traffic (e.g. not having to download the entire list for every

On this one point let me re-iterate how simple it was (certainly for me) to 
implement changedsince.

Also, we learned a HUGE lesson in CalDAV - initially there was no way for 
clients to quickly find which events in a calendar might have changed - 
instead they had to do full PROPFIND Depth:1 request - equivalent to a full 
listing of all events in the calendar. This quickly became a roadblock to 
scaleability and we developed the WebDAV Sync REPORT to allow clients to 
just get the list of changed events since the last time they sync'd. Pretty 
much all clients and servers now implement that.

Now its true that calendars can grow to be a lot larger than the overall 
set of time zones that exist today (some people have 10's of 1000s of 
events), so the need for sync is critical. But even with ~500 time zones in 
the IANA db, there is a significant gain to be had by using changedsince to 
cut down on bandwidth/power use for, e.g., limited mobile devices. As such 
I continue to beleive that changedsince is a critical piece for tzdist.

-- 
Cyrus Daboo

---

From: Mike Douglass <[email-masked]>
Date: Tue, 28 Oct 2014 10:22:34 -0400

I second (or third) that.

It is not hard to implement and does reduce traffic.

On 10/28/2014 10:16 AM, Cyrus Daboo wrote:
> On this one point let me re-iterate how simple it was (certainly for 
> me) to implement changedsince.
>
> Also, we learned a HUGE lesson in CalDAV - initially there was no way 
> for clients to quickly find which events in a calendar might have 
> changed - instead they had to do full PROPFIND Depth:1 request - 
> equivalent to a full listing of all events in the calendar. This 
> quickly became a roadblock to scaleability and we developed the WebDAV 
> Sync REPORT to allow clients to just get the list of changed events 
> since the last time they sync'd. Pretty much all clients and servers 
> now implement that.
>
> Now its true that calendars can grow to be a lot larger than the 
> overall set of time zones that exist today (some people have 10's of 
> 1000s of events), so the need for sync is critical. But even with ~500 
> time zones in the IANA db, there is a significant gain to be had by 
> using changedsince to cut down on bandwidth/power use for, e.g., 
> limited mobile devices. As such I continue to beleive that 
> changedsince is a critical piece for tzdist.
>

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 28 Oct 2014 10:48:19 -0400

Hi Stephen,

> http://martinfowler.com/articles/richardsonMaturityModel.html

Thanks for putting this together. I need to digest the content and 
discussion somewhat before diving in and updating the spec. Overall, what 
you propose looks good to me, though I might suggest some minor tweaks to 
names of paths (e.g., I think I prefer "zones" instead of "rules").

Also, I want to make use of the URI template specification (RFC 6570) to 
describe the various resources and to indicate optional prefixes (which I 
think would clarify some of Lester's issues). e.g.,

{/service-prefix}/capabilities

{/service-prefix,data-prefix}/zones{?changedsince}
{/service-prefix,data-prefix}/zones{?pattern}

{/service-prefix,data-prefix}/zones/{tzid}{?start,end}

{/service-prefix,data-prefix}/transitions/{tzid}{?start,end,changedsince}

Where {server-prefix} is the overall service root path, and {data-prefix} 
can be defined as follows:

empty: provides access to the latest versions of tz data from all 
publishers the tzdist services uses.

/publisher/{publisher} : provides access to the latest version of data from 
a specific publisher

/publisher/{publisher}/versions/{version}: provides access to specific 
version from a specific publisher

I think the base spec should only define the "empty" value behavior but 
make it clear that others can be defined in the future. Other resources 
could also be defined (e.g., an extension to lookup time zones in use at a 
specific location):

{/service-prefix,data-prefix}/location{?lat,long}

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Tue, 28 Oct 2014 17:50:33 +0000

On 28/10/14 14:16, Cyrus Daboo wrote:
> As such I continue to beleive that changedsince is a critical piece for
> tzdist.

I'd still like to understand how the race condition can be avoided if
one is only using the 4.2.2/4.2.3 sync process?

---

From: Stephen Colebourne <[email-masked]>
Date: Wed, 29 Oct 2014 13:35:10 +0000

> names of paths (e.g., I think I prefer "zones" instead of "rules").

zones/rules isn't something that matters too much one way or the other.

> empty: provides access to the latest versions of tz data from all publishers
> the tzdist services uses.

Just to note that given this definition, you can get identifier
clashes that you wouldn't get if publisher is mandatory. The up side
is one complete list of identifiers. Take your pick.

Stephen

---

From: Lester Caine <[email-masked]>
Date: Wed, 29 Oct 2014 14:27:25 +0000

> zones/rules isn't something that matters too much one way or the other.

Whilst I agree to a certain extent, I would tend towards 'rules' in the
first instance, allowing 'zones' and 'locations' to be expanded later
using the base rules which is all that is needed in the first instance.
OK probably 80% of those rules working from a 1970 cutoff will just be a
single time offset, going back in time, eventually those base rules will
combine with other historic rules and simple offsets to provide a more
complete picture for an area or single location.

This leaves a place open for a simple current tzdist server which only
has a small set of generic rules and all links to 'zones' or as aliases.

---

From: Lester Caine <[email-masked]>
Date: Wed, 29 Oct 2014 14:46:31 +0000

> has a small set of generic rules and all links to 'zones' or as aliases.
ARE as aliases.

i.e. rule is CET and all the 'zones' are aliases to that rule.