Subject: [Tzdist] tzdist examples

From: Cyrus Daboo <[email-masked]>
Date: Tue, 23 Dec 2014 10:54:31 -0500

Hi,
As discussed on the call last week I have put together a set of examples, 
showing what a client is expected to do in various scenarios in order to 
sync data (using the most recent proposal from the mail list with the new 
"synctoken", "datatoken", "publisher", and "version", information in the 
response). The goal here is to show that the latest proposal is sufficient 
to cover the cases of clients that start out with no time zone data, ones 
that start with existing time zone data provisioned from a non-tzdist 
source, and ones that have sync'd with one tzdist server and now have to 
sync with a different one.

1) Initial "full" sync: client with no cache of time zone data
	a) Client requests /capabilities from the server to determine supported 
features and data versions.
	b) Client does a /zones request (without changedsince) and caches the 
response.
	c) The client caches the response.
	d) The client determines which time zones it is interested in, and marks 
those for retrieval.
	e) For each time zone marked for retrieval, it downloads the time zone 
data via /zones{/tzid}.
	f) After all interesting time zones are downloaded it does a "partial" 
sync (see item #2). This allows the client to detect whether any time zones 
changed on the server whilst it was completing the "full" sync.
	
2) Subsequent "partial" sync: client with previously cached tzdist data
	a) Client does a /zones{?changedsince} request to find out what has change 
on the server (using the "synctoken" from the most recent list). If nothing 
is reported, the "partial" sync operation is complete
	b) For each item returned in the response:
		b.1) If the associated time zone is one that it previous cached the data 
for, it determines whether the "datatoken" in the response matches the 
previously cached version. If not, it marks that time zone for retrieval.
		b.2) It updates its cached list response with the new item from the 
latest response.
	c) For each time zone marked for retrieval, it downloads the time zone 
data via /zones{/tzid}.
	d) After all interesting time zones are downloaded it does a "partial" 
sync (see item 2). This allows the client to detect whether any time zones 
changed on the server whilst it was completing the "partial" sync.

3) Clients with existing time zone data from another tzdist server or 
provisioned by another mechanism. Note that in both cases the client has to 
know the publisher and version for every existing time zone. The client 
will do a modified version of the "full" sync (as per item #1) after which 
ordinary "partial" sync's (as per item #2) can be done.
	a) As per #1.a.
	b) As per #1.b.
	c) As per #1.c.
	d) The client matches the (publisher, version) information for each time 
zone in the response with the corresponding information in its existing 
time zone data cache. For each case where the publisher matches, but the 
version does not, it does the following:
		i) If the version on the tzdist server is older than the version cached 
locally, the local version will be preserved, and the corresponding tzdist 
response will be modified to update the version to match what is cached 
locally.
		ii) If the version on the tzdist server is the same as the version cached 
locally, the local version will be preserved.
		iii) If the version on the tzdist server is newer than the version cached 
locally, the time zone will be marked for retrieval.
        If the publisher does not match, but the time zone identifier does, 
the client chooses, at its own discretion, whether to retrieve the time 
zone data from the server in order to replace data from one publisher with 
the other.
	e) As per #1.e.
	f) As per #1.f.

-- 
Cyrus Daboo

---

From: Daniel Migault <[email-masked]>
Date: Tue, 23 Dec 2014 18:29:54 +0100

Hi Cyrus,

Thanks for providing these examples. As far as I remember all cases that
have been mentioned so far have been considered in this email.

I have added in text my questions.

BR,
Daniel

On Tue, Dec 23, 2014 at 4:54 PM, Cyrus Daboo <[email-masked]> wrote:

>         a) Client requests /capabilities from the server to determine
> supported features and data versions.
>         b) Client does a /zones request (without changedsince) and caches
> the response.
>         d) The client determines which time zones it is interested in, and
> marks those for retrieval.
>         e) For each time zone marked for retrieval, it downloads the time
> zone data via /zones{/tzid}.
>         f) After all interesting time zones are downloaded it does a
> "partial" sync (see item #2). This allows the client to detect whether any
> time zones changed on the server whilst it was completing the "full" sync.
>

My understanding of this scenario is that:
    - we consider that the client has previously retrieved the tz on a
given server, and sync with this specific server only.
    - synctoken is server specific, and cannot be used between different
servers. If so, we should also specify in the draft how the binding between
synctoken and server is performed. Do we use IP addresses, FQDN, other
server IDs...
    - the synctoken is updated anytime one of the tz hosted on the server
is updated.
    - datatoken is also server specific, and two different time zone on
different servers do not have the same datatoken.

        a) Client does a /zones{?changedsince} request to find out what has
> change on the server (using the "synctoken" from the most recent list). If
> nothing is reported, the "partial" sync operation is complete
>                 b.1) If the associated time zone is one that it previous
> cached the data for, it determines whether the "datatoken" in the response
> matches the previously cached version. If not, it marks that time zone for
> retrieval.
>                 b.2) It updates its cached list response with the new item
> from the latest response.
>         c) For each time zone marked for retrieval, it downloads the time
> zone data via /zones{/tzid}.
>         d) After all interesting time zones are downloaded it does a
> "partial" sync (see item 2). This allows the client to detect whether any
> time zones changed on the server whilst it was completing the "partial"
> sync.
>
>

> ordinary "partial" sync's (as per item #2) can be done.

I understand that version and publishers are those of the tz locally
retrieved on the client. Am I correct?

>         a) As per #1.a.
>         b) As per #1.b.
>         c) As per #1.c.
>         d) The client matches the (publisher, version) information for
> each time zone in the response with the corresponding information in its
> existing time zone data cache. For each case where the publisher matches,
> but the version does not, it does the following:
>                 i) If the version on the tzdist server is older than the
> version cached locally, the local version will be preserved, and the
> corresponding tzdist response will be modified to update the version to
> match what is cached locally.
>                 ii) If the version on the tzdist server is the same as the
> version cached locally, the local version will be preserved.
>                 iii) If the version on the tzdist server is newer than the
> version cached locally, the time zone will be marked for retrieval.
> _______________________________________________
> Tzdist mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tzdist
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 23 Dec 2014 18:24:54 +0000

While this is all very clever, and probably does solve the problem, it
isn't exactly simple or even that web-friendly.

One of the key benefits of ETags and defining resources that never
expire is that the full strength of internet caching can come into
play. The mechanism described below if the opposite - a proprietary
caching mechanism requiring additional work by the client and the
server. I'm not convinced this is right.

In order to make maximum use of internet caching (a really Good Thing
particularly when polling is being planned as a usage model), the
design needs to focus on making ETags work. ETags are basically simple
- for any individual response it is necessary to construct a unique
ETag for the data, typically some form of hash, but an internal
database identifier would be equally applicable.

If the response from "/zones/CET" has ETag 12345, then clients can
know that the data hasn't changed so long as the ETag of the resource
is still 12345. The same applies for a list response - "/zones" -
which might have an ETag 45678. So long as it is still 45678, no
change has occurred.

When no data has changed, the response to /zones will not change. The
ETag will remain the same, and clients merely need to perform a
standard ETag based "If-None-Match" query.

When a new monolithic release occurs, the response to "/zones" will
change, and the ETag changes too, say from 45678 to 87654. Each client
that polls will then pull back the new response and process it.

On first pull, this involves deciding which zones they are interested
in. Each of those zones will then be retrieved, such as the
"/zones/CET" response with ETag 12345.

On a subsequent pull, the client can decide if any additional zones
have been added. It can then request each of the zones it is
interested in, with the ETag of the zone avoiding unecessary data
transfer.

As described below, the client can repeat this procedure once it has
completed it to ensure that no change occurred while the client was
processing (the race condition).

When a new individual zone change is made (non-monolithic update), the
same applies as above. Nothing different (which implies that "/zones"
must change ETag when one of its child zones changes). The only
overhead is that the "/zones" response is bigger than it might have
been with changedsince. The ETag caching takes care of not downloading
unchanged individual zones.

Given the above, I don't see any need for "changedsince" or "datatoken".

One final (optional) optimisation is that the ETag of the individual
zone can be returned in the list response. This is effectively the
"datatoken", but gives it a much more meaningful definition. Since
making use of this requires querying the ETag from the clients HTTP
cache, not all clients will be able to make use of it, but using the
ETag of the zone response here is much more web-friendly than a second
random "datatoken".

Section 3, matching {publisher, version} from two different servers,
seems sound enough to me for the more complex use cases.

Stephen

On 23 December 2014 at 15:54, Cyrus Daboo <[email-masked]> wrote:
> know the publisher and version for every existing time zone. The client will
> do a modified version of the "full" sync (as per item #1) after which
>         d) The client matches the (publisher, version) information for each
> time zone in the response with the corresponding information in its existing
> the client chooses, at its own discretion, whether to retrieve the time zone
> data from the server in order to replace data from one publisher with the
> other.
>         e) As per #1.e.
>         f) As per #1.f.
>
>
> --
> Cyrus Daboo
>

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 23 Dec 2014 13:45:58 -0500

Hi Stephen,

> ETag of the zone response here is much more web-friendly than a second

So I am happy with changing "datatoken" to "etag" in the list response (in 
reality I was probably going to use the resource etag as the datatoken in 
my implementation anyway).

However, I still feel strongly that changedsince is important to reduce the 
response size for the "partial" sync case. Right now there are something 
like 500 possible time zones that can be returned for the IANA database. 
That could easily expand to many more (e.g., someone suggested that the 
/posix and /right set could also be included which would take use to 1500). 
What is more, if the publisher uses an "incremental version" model, it is 
likely that they may be issuing small, frequent updates to address a change 
as soon as it occurs. Note that a client isn't required to use changedsince 
- if it thinks life is easier just geting the full response everytime, then 
of course it is free to do so. But those clients that prefer the smaller 
response for the "partial" sync, do have that option available.

-- 
Cyrus Daboo

---

From: Stephen Colebourne <[email-masked]>
Date: Tue, 23 Dec 2014 19:09:05 +0000

On 23 December 2014 at 18:45, Cyrus Daboo <[email-masked]> wrote:
> reality I was probably going to use the resource etag as the datatoken in my
> implementation anyway).

I don't know how easy it will be for clients to use, but its
definitely best to spec it as being the etag.

> response for the "partial" sync, do have that option available.

Bear in mind that if each client polls once an hour, that will result
in a lot of extra work for the server as compared to an ETag of the
whole response which can be cached in the internet (or low-level HTTP
stack of the server). As such, I'm not convinced it will actually make
things better. If you can get an ETag to work for
"/zones?changedsince" then you are sorted, but I don't know offhand if
that works.

Stephen

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 23 Dec 2014 14:48:54 -0500

Hi Stephen,

> "/zones?changedsince" then you are sorted, but I don't know offhand if

I did come across this draft which comes close to something like that:

<http://tools.ietf.org/html/draft-nottingham-http-patch-status-00>

However, that is now expired and I am not sure whether it will progressed 
(and at this point we don't want to add new dependencies on tzdist). Plus 
it is not cache friendly in that it explicitly states that the resulting 
patch document is not cacheable.

That sais, I see no reason why a client cannot use etags with either the 
"full" or "partial" list action request. Note, as per the Note in Section 
4.2.2 of RFC 7234 (HTTP caching) it appears that many caches do include the 
URI query parameters as part of the "freshness" determination. So, that 
means the first client to so a request of /zones?changedsince=rev1 will 
cause an intermediate cache to be populated. If that client repeats the 
request, or another client does the same request, then it will be served 
from the cache. What is more, the first client could use ETag value from 
its original request in an If-None-Match header with the subsequent request 
and get back a 304. The resource will change when the server refreshes a 
change from the publisher. After that, clients will start using 
/zones?changedsince=rev2 etc and that too can be cached.

So it is my belief that ETags will work with both the "full" and "partial" 
list actions in tzdist. We can emphasize that behavior by adding suitable 
text to Section 4.1.4 of tzdist where we already talk about use of ETags 
for the get action. We can extend that to have clients also use ETags for 
the list action polling.

-- 
Cyrus Daboo

---

From: Paul Eggert <[email-masked]>
Date: Tue, 23 Dec 2014 12:42:28 -0800

Cyrus Daboo wrote:
> I still feel strongly that changedsince is important to reduce the response size
> for the "partial" sync case. Right now there are something like 500 possible
> time zones that can be returned for the IANA database. That could easily expand
> to many more (e.g., someone suggested that the /posix and /right set could also
> be included which would take use to 1500). What is more, if the publisher uses
> an "incremental version" model, it is likely that they may be issuing small,
> frequent updates to address a change as soon as it occurs.

ETags should suffice for all that, no?  I'm still not seeing what sigificant 
thing changesince buys us that ETags do not.  For metadata, if a "please list 
the catalog" response comes with an ETag, the client can simply use that in its 
next "please list the catalog" request, instead of a changedsince.  Similarly 
for data.

>  That could easily expand to many more (e.g., someone suggested that the /posix and /right set could also be included which would take use to 1500).

"/posix" is the same as the default; there should be no reason to supply both. 
(Typically "posix" just symlinks to the default, and tzdist shouldn't insist 
that they be copies.)  "/right" is only for clients that care about leap 
seconds, and there should be a way for thinner clients to list just the non-leap 
data (or just the leap data, if that's what they really want).  If all this is 
important (which is not clear), then it can be addressed by giving clients the 
ability to do the equivalent of "ls posix" and "ls right" on the server side. 
That would work well with ETags, and there should be no need for adding the 
confusing complexity of "changedsince" and or "version" to the protocol.

I quite agree with Stephen Colebourne that we should be using ETags and not 
reinventing its wheel.

---

From: Doug Royer <[email-masked]>
Date: Tue, 23 Dec 2014 15:21:26 -0700

On 12/23/2014 08:54 AM, Cyrus Daboo wrote:
> Hi,
> As discussed on the call last week I have put together a set of 
> examples, ..

That covers everything I need. Thanks!

-- 

Doug Royer - (http://K7DMR.us / http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 23 Dec 2014 18:25:28 -0500

Hi Paul,

--On December 23, 2014 at 12:42:28 PM -0800 Paul Eggert 
<[email-masked]> wrote:

>> I still feel strongly that changedsince is important to reduce the
>> /posix and /right set could also be included which would take use to
>> 1500). What is more, if the publisher uses an "incremental version"
>> model, it is likely that they may be issuing small, frequent updates to
>> address a change as soon as it occurs.
>
> ETags should suffice for all that, no?  I'm still not seeing what
> sigificant thing changesince buys us that ETags do not.  For metadata, if
> a "please list the catalog" response comes with an ETag, the client can
> simply use that in its next "please list the catalog" request, instead of
> a changedsince.  Similarly for data.

ETag does not allow for just the set of changed time zone meta-data to be 
returned in list action. So in the case of incremental version updates 
clients end up having to get the entire set of meta-data rather than just 
that for those time zones that have changed.

So let me clarify by trying to compare the ETag vs changedsince behavior 
for monolithic vs incremental version publishers:

1) Full sync /zones
    a) monolithic publisher: returns 500 items
    b) incremental publisher: returns 500 items

2) Poll to sync /zones + If-None-Match - nothing changed
    a) monolithic publisher: returns HTTP 304 status
    b) incremental publisher: returns HTTP 304 status

3) Poll to sync /zones?changedsince - nothing changed
    a) monolithic publisher: returns empty list response
    b) incremental publisher: returns empty list response

4) Poll to sync /zones + If-None-Match - assume one time zone changed
    a) monolithic publisher: returns 500 items
    b) incremental publisher: returns 500 items

5) Poll to sync /zones?changedsince - assume one time zone changed
    a) monolithic publisher: returns 500 items
    b) incremental publisher: returns 1 item

So 5(b) is a win for reducing the size of the response in the case where a 
small number of time zones change when using an incremental publisher. Note 
that all of the above options are supported by the protocol as it stands 
today and can make use of intermediary caching for improved performance 
during polling.

Whilst the IANA DB is currently using a monolithic version update scheme, 
someone else could decide to republish that data using an incremental 
scheme, e.g., by introducing their own per-time zone revision counters.

So I don't want to preclude the possibility of 5(b). So the question is can 
we get consensus on whether or not the optimization that 5(b) represents is 
worthwhile for the base tzdist specification. The answer to that determines 
whether changessince stays or goes...

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Wed, 24 Dec 2014 13:51:08 +0100

Difficult to quote inline from here. Etag can be used to confirm a data block has already been sent, but need changedsince to ONLY return a list of changes. For historic material it is essential that we have a reliable published source ... complete with ALL ... the change set changes ... so that we can rely on the TZ data being used matching the archive material. It is essential that this is generaly available from multiple sources, and for that reason 'version' is as critical as having a complete data set rather than something randomly accurate prior to 1970. Using changedsince on a reliable historic publisher will NOT be server specific so that as servers change over time we can maintain a reliable source. TZ has not proven to be reliable for this purpose and neither will tzdist if it is no possible to recover when a server goes off line.
Being able to establish what has changed version to version is what is currently missing. We could do that in the client, but a new user of historic material needs to be able to sync with the same data that was used at some point in the past. That may simply be a download of everything over time, but need only be a base + changes since ... which changedsince provides nicely.
I think the examples do cover my use, but need to work threw when back in the office.

Sent from my android device so the quoting is crap!

---

From: Daniel Migault <[email-masked]>
Date: Wed, 24 Dec 2014 15:20:59 +0100

Dera Friends and Colleagues,

As explained by Cyrus, "changesince" has an advantage over ETag as it does
not only inform a change has occured, but also enable to apply filters on
the response which results in lower response. It only apply when versioning
happens in an incremental mode.
On the other hand using "changesince" may makes caching not so effective,
and makes code more complex.

In order to decide whether "changesince" needs to be considered, we would
like the WG to say ASAP:
    a) Reducing response size for incremental mode should be considered in
teh current spec.
    b) Reducing response size for incremental should be left for future
work

My personal opinion is b). "changesince" may be introduced later as an
optimization, so we can ignore this case for now.

For the  next version of the draft I also think that using the terminology
synctoken, datatoken is very useful to describe the problem. Even though
ETag may become the way these tokens are implemented. I believe it would be
useful to have the problem description be part of the draft to show ETag
covers all our use cases.

Here is the example from Cyrus:

The advantage of using changesince over ETag is that
4) Poll to sync /zones + If-None-Match - assume one time zone changed
   a) monolithic publisher: returns 500 items
   b) incremental publisher: returns 500 items

On Wed, Dec 24, 2014 at 12:25 AM, Cyrus Daboo <[email-masked]> wrote:

> Hi Paul,
>
> --On December 23, 2014 at 12:42:28 PM -0800 Paul Eggert <
> [email-masked]> wrote:
>
> So I don't want to preclude the possibility of 5(b). So the question is
> can we get consensus on whether or not the optimization that 5(b)
> represents is worthwhile for the base tzdist specification. The answer to
> that determines whether changessince stays or goes...
>
> --
> Cyrus Daboo
>
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Stephen Colebourne <[email-masked]>
Date: Wed, 24 Dec 2014 14:27:33 +0000

I don't see any major issue in deferring.

I do believe it is possible to design a URL based approach to
changedsince along these lines:

/zoneschangedsince/{synctoken}

where calling that resource would return the same response as the
/zones?changedsince={synctoken} but be fully ETag cachable. The
original /zones request would return the "changes since" URL as part
of the response if it supported it.

I don't love it as a URL design, but the tensions of trying to spot
partial updates of a largish dataset are slightly unusual.

Stephen

On 24 December 2014 at 14:20, Daniel Migault <[email-masked]> wrote:
>     b) Reducing response size for incremental should be left for future work
>
>> can we get consensus on whether or not the optimization that 5(b) represents
>> is worthwhile for the base tzdist specification. The answer to that
>> determines whether changessince stays or goes...
>>
>> --
>> Cyrus Daboo
>>
>>

---

From: Lester Caine <[email-masked]>
Date: Wed, 24 Dec 2014 17:33:01 +0100

If tzdist is not providing a service that allows selection of sub sets of data be that updated TZids or a truncated set of data is there any point to it? If we have the TZ raw data reliably available then we have the full set anyway. 
tzdist is essentially an incremental system with either 'expand' or just update information. Downloading a full set of data is only a backup if the data is not already cached by the client ... by what ever means ... 
You ask tzdist simply if there is a curent update that may affect things, otherwise the client downloads nothing.

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 24 Dec 2014 13:21:49 -0500

Hi Lester,

--On December 24, 2014 at 5:33:01 PM +0100 Lester Caine 
<[email-masked]> wrote:

> If tzdist is not providing a service that allows selection of sub sets of
> data be that updated TZids or a truncated set of data is there any point
> to it? If we have the TZ raw data reliably available then we have the
> full set anyway.  tzdist is essentially an incremental system with either
> 'expand' or just update information. Downloading a full set of data is
> only a backup if the data is not already cached by the client ... by what
> ever means ...  You ask tzdist simply if there is a curent update that
> may affect things, otherwise the client downloads nothing.

And that is how the current protocol works and if we dropped changedsince, 
it would work that way too.

You need to distinguish between downloading "meta-data" vs "time zone data" 
- that later being the actual time zone rules in the chosen data format. As 
it stands, clients always download some amount of meta-data and from that 
decide what time zone data to retrieve. So when the data for only one time 
zone changes, clients will do one request to get the meta-data and then one 
request to get the new time zone data. There is no need to download a full 
set of data.

The only question is how much of the meta-data do we want the client to 
download? Without changedsince, clients would be sent all the meta-data, 
regardless of how much has changed.

I believe this is all adequately described by the examples I posted. If 
not, please point to specific parts that need clarification and I will 
endeavor to provide that.

-- 
Cyrus Daboo

---

From: Lester Caine <[email-masked]>
Date: Wed, 24 Dec 2014 21:00:05 +0100

Cyrus - will need to digest when I get back home in the new year ... only able to digest little bits on the phone. I think we both have the same mechanism but for different jobs. Only difference is I expect client to have full set of data to handle historic documents. If a client only has a few TZid's then they may need a lot more when processing that data.

---

From: Daniel Migault <[email-masked]>
Date: Fri, 26 Dec 2014 18:45:50 +0100

My current understanding is that there is not a strong support for keeping
"changedsince" in the current spec, and that it could be left for future
work.

BR,
Daniel

On Wed, Dec 24, 2014 at 9:00 PM, Lester Caine <[email-masked]> wrote:

> Cyrus - will need to digest when I get back home in the new year ... only
> able to digest little bits on the phone. I think we both have the same
> mechanism but for different jobs. Only difference is I expect client to
> have full set of data to handle historic documents. If a client only has a
> few TZid's then they may need a lot more when processing that data.
>
>

-- 
Daniel Migault
Orange Labs -- Security
+33 6 70 72 69 58

---

From: Ken Murchison <[email-masked]>
Date: Mon, 29 Dec 2014 08:48:36 -0500

On 12/26/2014 12:45 PM, Daniel Migault wrote:
> My current understanding is that there is not a strong support for 
> keeping "changedsince" in the current spec, and that it could be left 
> for future work.

I wouldn't go that far.

I agree with Cyrus that changedsince is important to reduce the size of 
the list response for clients looking to update the full set of time 
zones.  This is especially true now that we have added versioning 
information to the list response thus growing its size.

Regardless of whether a client chooses to use changedsince, this 
functionality is fairly trivial to implement in a server, and 3 existing 
servers have already done so.  In fact, this functionality has been 
interop tested doing server to server time zone updates.

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University