Subject: [Sedate] Offset and timezone: an invitation to regretable error

From: Watson Ladd <[email-masked]>
Date: Thu, 11 Nov 2021 12:51:34 -0800

Dear sedate,

I really, really want to emphasize how utterly harmful offsets are in
times. Offsets do not give the right answer when asking the question
"what time is one day from now"? We just saw this disrupt a side
meeting. To the extent that we need to disambiguate when DST
replicates times, we're fundamentally asking a confusing question, and
should be very careful.

Having both offset and timezone, and then marking which one actually
matters, is an invitation to disaster. Implementations will use the
information in them incorrectly if offered a choice. An implementation
that only supports timezone, but not the selector, although preserving
it, will break.

I also want to strongly question the idea of display hints in times,
particularly for calendrical calculations. A month does not have the
same length in different calendars. To get Rosh Chodesh displayed
correctly is not going to happen with a hint to display an event every
month in the Hebrew calendar: the month used to compute the repetition
must be the Hebrew month.

My suggestion is the following: we use year-month-day
hour-minute-second+disambiguator and a mandatory locale. Locale is a
tzdb timezone and a calendar. Disambiguator is A or B for the first or
second repetition of the time, only permissable when the rest of the
representation is doubled. This format is always bijective to UTC for
a given locale.

Sincerely,
Watson Ladd
-- 
Astra mortemque praestare gradatim

---

From: Ujjwal Sharma <[email-masked]>
Date: Fri, 12 Nov 2021 15:35:18 +0530

Hi Watson!

> replicates times, we're fundamentally asking a confusing question, and

I agree. Offset timestamps (we call them Instants in Temporal) are great 
for identifying a single instant on the timeline, but to do any useful 
arithmetic correctly above hours, you need to know the IANA timezone to 
be used as the context. One day is not 24 hours, it can be 23 or 25 and 
the arithmetic needs to take that into account.

> that only supports timezone, but not the selector, although preserving

During the discussion with Pete, we came up with a number of use-cases 
why it is useful in some cases to have both pieces of information, I 
hope Pete sounds out those notes soon so we could use them as reference.

> must be the Hebrew month.

I agree. In Temporal, the calendar annotation is not just use for 
internationalization, but also for arithmetic. When you take a Hebrew 
date and add a month to it, the result of this operation is 
calendar-specific.

> representation is doubled. This format is always bijective to UTC for

I'd love to hear what people think about the mandatory part here. In our 
case, we consider UTC to be the default timezone and the ISO (proleptic 
gregorian) to be the default calendar in case they're not included.

Ujjwal

-- 
Ujjwal "Ryzokuken" Sharma (he/him)

Compilers Hacker, Node.js Core Collaborator and Speaker

---

From: Daniel Migault <[email-masked]>
Date: Fri, 12 Nov 2021 08:43:05 -0500

On Fri, Nov 12, 2021 at 5:05 AM Ujjwal Sharma <[email-masked]> wrote:

> I am curious to see these use-cases.

> I think I pretty much share Watson's  idea where we have a single
deterministic way to interpret the date format. This results in removing
fields that may be interpreted differently by implementations. That there
is a default value is not problematic in my opinion as long as everyone has
the same default values. This could be performed by the application or in
the format.

> Sedate mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sedate
>

-- 
Daniel Migault
Ericsson

---

From: Pete Resnick <[email-masked]>
Date: Fri, 12 Nov 2021 08:51:13 -0600

> hope Pete sounds out those notes soon so we could use them as 

Here's what I got:

1. The "UI Case": In this case, what I mean is "at a particular UTC 
offset for a single time instance", but I want to give some textual info 
that can be displayed to the user instead of some numeric offset. So the 
offset is "what I mean", and the zone name is the "hint".

2. The "Buggy Sender Database Case": Here, "what I mean" is definitely 
the zone name. However, I don't realize that I have an out of date TZ 
database in my possession. If I include the UTC offset as a "hint", my 
recipient might notice that something is amiss and either helpfully 
correct me or reject the time if it is rendered ambiguous.

3. The "Missing or Buggy Receiver Database Case": In this case, we're 
thinking of a receiver of a time that is too stupid to have an 
up-to-date (or any) proper TZ database of their own where, at least for 
a single instance, having the UTC offset as the "hint" will allow the 
recipient to do the right thing, or when the receiver has the buggy TZ 
DB, similar to the previous case.

4. The "Missing DST Instance Case": When I've got only a zone name and 
the local time to represent a particular instance of time, like 
"2021-11-07T01:30:00 America/Chicago", I'm missing a key piece of 
information: Is that 1:30am before or after Daylight Saving Time has 
ended? It could mean 2021-11-07T01:30:00-05:00 or it could mean 
2021-11-07T01:30:00-06:00, because on that particular day, 1:30am occurs 
twice. So for a single instance, "what I mean" is the offset and the 
"hint" is the zone name. For other than a single instance, I probably 
mean the zone name, but I need additional context if it's going to 
overlap the DST switch.

I generally agree with Watson that two pieces of potentially conflicting 
data are not a good thing, but the world is full of imperfect situations 
where some additional data, **with well-described semantics**, might be 
a good thing.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Pete Resnick <[email-masked]>
Date: Fri, 12 Nov 2021 08:55:18 -0600

Oh, and just to be clear, in each case I'm making the distinction 
between "what I mean" as one piece of data and "a hint to be helpful to 
the reciever" as the other piece of data. In each case, those two things 
might be different.

[r

On 12 Nov 2021, at 8:51, Pete Resnick wrote:

>> During the discussion with Pete, we came up with a number of 
>> use-cases why it is useful in some cases to have both pieces of 
>> information, I hope Pete sounds out those notes soon so we could use 
>> them as reference.
>
> Here's what I got:
>
> offset for a single time instance", but I want to give some textual 
> info that can be displayed to the user instead of some numeric offset. 
> So the offset is "what I mean", and the zone name is the "hint".
>
> up-to-date (or any) proper TZ database of their own where, at least 
> for a single instance, having the UTC offset as the "hint" will allow 
> the recipient to do the right thing, or when the receiver has the 
> buggy TZ DB, similar to the previous case.
>
> 2021-11-07T01:30:00-06:00, because on that particular day, 1:30am 
> occurs twice. So for a single instance, "what I mean" is the offset 
> and the "hint" is the zone name. For other than a single instance, I 
> probably mean the zone name, but I need additional context if it's 
> going to overlap the DST switch.
>
> I generally agree with Watson that two pieces of potentially 
> conflicting data are not a good thing, but the world is full of 
> imperfect situations where some additional data, **with well-described 
> semantics**, might be a good thing.
>
> pr

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 12 Nov 2021 16:56:17 +0100

Hi Pete,

Thank you for writing up the use case side.

With a greenfield, we’d know what to do.
We do have a brownfield, though, so the “what I mean” = UTC-referenced time is occupied by RFC 3339 or an extension of that.

The interesting part of your use cases that I didn’t have on the radar yet is that in some cases, *detecting* a mismatch is the most important objective, and this is not just about X overrides Y or vice versa.

Grüße, Carsten

> On 2021-11-12, at 15:55, Pete Resnick <resnick=[email-masked]> wrote:
> 
> Oh, and just to be clear, in each case I'm making the distinction between "what I mean" as one piece of data and "a hint to be helpful to the reciever" as the other piece of data. In each case, those two things might be different.
> 
> [r
> 
>>> During the discussion with Pete, we came up with a number of use-cases why it is useful in some cases to have both pieces of information, I hope Pete sounds out those notes soon so we could use them as reference.
>> 
>> Here's what I got:
>> 
>> 1. The "UI Case": In this case, what I mean is "at a particular UTC offset for a single time instance", but I want to give some textual info that can be displayed to the user instead of some numeric offset. So the offset is "what I mean", and the zone name is the "hint".
>> 
>> 2. The "Buggy Sender Database Case": Here, "what I mean" is definitely the zone name. However, I don't realize that I have an out of date TZ database in my possession. If I include the UTC offset as a "hint", my recipient might notice that something is amiss and either helpfully correct me or reject the time if it is rendered ambiguous.
>> 
>> 3. The "Missing or Buggy Receiver Database Case": In this case, we're thinking of a receiver of a time that is too stupid to have an up-to-date (or any) proper TZ database of their own where, at least for a single instance, having the UTC offset as the "hint" will allow the recipient to do the right thing, or when the receiver has the buggy TZ DB, similar to the previous case.
>> 
>> 4. The "Missing DST Instance Case": When I've got only a zone name and the local time to represent a particular instance of time, like "2021-11-07T01:30:00 America/Chicago", I'm missing a key piece of information: Is that 1:30am before or after Daylight Saving Time has ended? It could mean 2021-11-07T01:30:00-05:00 or it could mean 2021-11-07T01:30:00-06:00, because on that particular day, 1:30am occurs twice. So for a single instance, "what I mean" is the offset and the "hint" is the zone name. For other than a single instance, I probably mean the zone name, but I need additional context if it's going to overlap the DST switch.
>> 
>> I generally agree with Watson that two pieces of potentially conflicting data are not a good thing, but the world is full of imperfect situations where some additional data, **with well-described semantics**, might be a good thing.
>> 
>> pr
> 
> 
> --

---

From: "Bron Gondwana" <[email-masked]>
Date: Mon, 15 Nov 2021 15:04:58 +1100

Thanks for writing this up Pete.

I'm going to intersperse my commentary on each use case, as well as address a couple of other things that came up.

> offset is "what I mean", and the zone name is the "hint".

There's definitely some "affordance to the UI" and also "affordance to the person reading the raw representation" to consider here.  Already `2022-11-25T22:00:00Z` and `2022-11-26T09:00:00+11:00` mean the same instant in time, one just has some additional context which is that you probably expect it to be 9am local for the person dealing with it, and for the day to be Saturday for them as well.  Not that "Saturday" is encoded in there, only the fact that it's the 26th locally.

Meanwhile, I know that Pete wrote this email on Friday morning his time (probably, unless his system is confused) because the email I'm replying to says:

Date: Fri, 12 Nov 2021 08:51:13 -0600

And that does contain useful context that `2021-11-12T14:51:13Z` would not have.

But yes, the zone name is a hint in the same way that presenting the time in local rather than UTC is a hint.

> correct me or reject the time if it is rendered ambiguous.

This is surprisingly common when sending dates between heterogeneous systems.  I have debugged quite a lot of calendar issues in my time due to this problem.  iCalendar kinda works around this by shipping an entire (or at least: truncated to now and future) timezone definition.  A calendar event for Melbourne right now might have this monstrosity at the top:

BEGIN:VTIMEZONE
TZID:Australia/Melbourne
LAST-MODIFIED:20210816T175139Z
X-LIC-LOCATION:Australia/Melbourne
TZUNTIL:20211114T230000Z
BEGIN:STANDARD
TZNAME:AEST
TZOFFSETFROM:+1100
TZOFFSETTO:+1000
DTSTART:20080406T030000
RRULE:FREQ=YEARLY;BYMONTH=4;BYDAY=1SU
END:STANDARD
BEGIN:DAYLIGHT
TZNAME:AEDT
TZOFFSETFROM:+1000
TZOFFSETTO:+1100
DTSTART:20081005T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=1SU
END:DAYLIGHT
END:VTIMEZONE

Which takes up more bytes than the event itself!

But part of the reason for that is that there isn't a globally synchronised database of timezone identifiers that devices actually get in a timely manner.  It's better now than it was even a few years ago, but timezones are still highly political and timezone databases are updated by operating systems on their own update schedule, so out of date copies are common.

So just a zone name without an offset hint is likely to cause synchronisation errors.  The fix might need to be "prompt the user to get a fixed value" if the two don't align, but the alternative isn't "everything is good" - the alternative is "*you have no hint that something might be wrong unless you discover the issue out-of-band".*

> DB, similar to the previous case.

There's a somewhat related case here, which is "the expensive sort".  Sorting dates by their UTC representation is quite cheap when you have offsets already.  You can get a fast, stable sort with an offset hint.  Even though the sort may not be exactly correct, it's going to a lot more correct than the sort you'd get if you only consider the floating time without the offset/zone at all, and a ton cheaper than resolving every timezone to known what the offset should be for every date.

This also comes back to the "human reading one of these" - if the human doesn't know where in the world the zone is, they can still calculate from the offset hint a human guess as to where this sits in UTC time.  It's a human affordance which could be useful.

FInally of course on this, it's the serialization of data model which contains both a zone name and an offset - which is useful for recreating state reliably at the other end.  You can then validate that state to see if you still think it's right.

> mean the zone name, but I need additional context if it's going to 

This is one that can also be dealt with a "standard/daylight" flag in the data structure rather than an offset.  My only argument here is that the offset serves the same purpose and you don't need both - and I believe the offset adds enough value in other places.

> where some additional data, **with well-described semantics**, might be 

I likewise agree that in a perfect world where politically zones were all agreed upon, and everybody updated their definitions at wire speed (like: DNS for example), and humans know the zone mappings in their heads for reading dates, then we wouldn't need this (except for the sorting case, where I still think having the offset data cached right in the structure is useful).  But we don't live in that world.

I also want to address two other things which came up in discussion:

*1) what about past dates where we might know local time by somebody's but not know what zone we were in reliably (e.g. in a warzone).*

Two responses to that.

a) you're screwed regardless - you can't get a perfectly accurate mapping to an exact UTC - so you either take a best guess as to UTC or you just call it floating time without a zone.  It's not like you could represent this in RFC3339 format before either.

b) my example "zone name is primary, offset is hint" works fine for this - as fine as for future dates where the zone might change.  If you get better information about what UTC offset was in place later, you can re-resolve the date - which you couldn't do with an RFC3339 production where the context was lost.

*2) what about using GEO locations instead of zone names?**
*

This debate has been had many times over.  It's a significant privacy consideration, so you'd wind up giving the offsets of the middle of the zone, or the middle of some large bounding box that you expected would always be inside the zone still.

Suddenly you've just recreated zone names, but harder to read and easier to get wrong.  And if different software chooses different magic points inside the zone - you're also adding signal to help fingerprint individual users.  There's an argument for moving away from the current named zones to UUIDs and an additional mapping layer (it gets raised over on tzdist every so often) - which is good from a "take the politics out of zone names" context, but it makes it even worse for human readability.  e.g.

`2022-11-26T09:00:00[uuid/839b6735-d083-4b15-960c-bdf7b1deb8ce]`

Good luck figuring out when THAT is in UTC when reading a protocol trace without an almanac in your other hand!

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]