321653 - import new TZ database

Status: VERIFIED FIXED

(Calendar :: Internal Components, defect, P1)

Description

[Dan Mosedale (:dmosedale, :dmose)]

The existing TZ database doesn't have the US timezone changes passed by the legislature in 2005.

Comment 1

[Reed Loden [:reed]]

The bugspam monkeys have struck again. They are currently chewing on default assignees for Calendar. Be afraid for your sanity!

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

The changes mentioned take effect in Spring 2007, and so attempting to look at the agenda that far out to check conflicts with recurring events is going to present the wrong data to the user.  Given that people sometimes plan things like (e.g.) trips with airplane flights many months in advance, I think we need this.

Comment 3

[Matthew (lilmatt) Willis]

Attachment: new timezone db

This was generated using vzic 1.3 and the latest available Olson db: 2006k

Comment 4

[Stefan Sitter [:ssitter]]

I don't know the internal tz handling therefore I have to ask:

I see that all the existing timezone definitions "/mozilla.org/20050126_1/*" will be replaced by new definitions "/mozilla.org/20060913_1/*".

What happens if Sunbird/Lightning hits one of the many existing events or tasks that use the old tzid? Will they be mapped? Will they be handled like unknown timezones? Will they be lost?

In addition a grep for "20050126_1" shows other files that use this string to create/identify/whatever this timezones.

Comment 5

[Matthew (lilmatt) Willis]

The following timezones were added by this Olson db update:

"/mozilla.org/20060913_1/America/Argentina/Buenos_Aires",
"/mozilla.org/20060913_1/America/Argentina/Catamarca",
"/mozilla.org/20060913_1/America/Argentina/Cordoba",
"/mozilla.org/20060913_1/America/Argentina/Jujuy",
"/mozilla.org/20060913_1/America/Argentina/La_Rioja",
"/mozilla.org/20060913_1/America/Argentina/Mendoza",
"/mozilla.org/20060913_1/America/Argentina/Rio_Gallegos",
"/mozilla.org/20060913_1/America/Argentina/San_Juan",
"/mozilla.org/20060913_1/America/Argentina/Tucuman",
"/mozilla.org/20060913_1/America/Argentina/Ushuaia",
"/mozilla.org/20060913_1/America/Atikokan",
"/mozilla.org/20060913_1/America/Bahia",
"/mozilla.org/20060913_1/America/Blanc-Sablon",
"/mozilla.org/20060913_1/America/Campo_Grande",
"/mozilla.org/20060913_1/America/Danmarkshavn",
"/mozilla.org/20060913_1/America/Indiana/Petersburg",
"/mozilla.org/20060913_1/America/Indiana/Vincennes",
"/mozilla.org/20060913_1/America/Moncton",
"/mozilla.org/20060913_1/America/North_Dakota/Center",
"/mozilla.org/20060913_1/America/North_Dakota/New_Salem",
"/mozilla.org/20060913_1/America/Toronto",
"/mozilla.org/20060913_1/Antarctica/Rothera",
"/mozilla.org/20060913_1/Asia/Choibalsan",
"/mozilla.org/20060913_1/Asia/Chongqing",
"/mozilla.org/20060913_1/Asia/Macau",
"/mozilla.org/20060913_1/Asia/Makassar",
"/mozilla.org/20060913_1/Asia/Oral",
"/mozilla.org/20060913_1/Asia/Qyzylorda",
"/mozilla.org/20060913_1/Asia/Sakhalin",
"/mozilla.org/20060913_1/Australia/Currie",
"/mozilla.org/20060913_1/Europe/Guernsey",
"/mozilla.org/20060913_1/Europe/Isle_of_Man",
"/mozilla.org/20060913_1/Europe/Jersey",
"/mozilla.org/20060913_1/Europe/Mariehamn",
"/mozilla.org/20060913_1/Europe/Volgograd",


The following timezones are no longer in the Olson db:
/mozilla.org/20060913_1/Africa/Timbuktu
/mozilla.org/20060913_1/America/Buenos_Aires
/mozilla.org/20060913_1/America/Catamarca
/mozilla.org/20060913_1/America/Cordoba
/mozilla.org/20060913_1/America/Indianapolis
/mozilla.org/20060913_1/America/Jujuy
/mozilla.org/20060913_1/America/Louisville
/mozilla.org/20060913_1/America/Mendoza
/mozilla.org/20060913_1/America/Rosario
/mozilla.org/20060913_1/Asia/Chungking
/mozilla.org/20060913_1/Asia/Macao
/mozilla.org/20060913_1/Asia/Ujung_Pandang
/mozilla.org/20060913_1/Europe/Belfast


Note that a few of these are renames, such as the /America/Argentina/* ones

Comment 6

[Matthew (lilmatt) Willis]

This does bring up some good points.

- Should we automatically migrate a user's timezone preference from a 20050126_1 version to a 20060913_1 version if one exists? If so that implies a mapping function somewhere.  Perhaps we shouldn't, as timezones that change (such as from the US Daylight Savings change) will then start interpreting dates differently, perhaps causing dataloss.

- If we leave their pref as is, it _won't_ be found in the menulist/listbox if and when they go to select their timezone again, which will cause an error.


More stuff to think about.

Comment 7

[Matthew (lilmatt) Willis]

Attachment: rev1 - updates timezones

To prevent string changes, I hardcoded any new timezones using the English string, and marked them with an "XXX" comment.  For timezones that were simply renamed, I used their previous entity key, and marked them with an "XXX" comment.  I removed any timezones that were removed from the Olson db.


This doesn't answer ssitter's or my questions, so it probably shouldn't be checked in even after r1/r2 until dmose or mvl say so.

Comment 8

[Matthew (lilmatt) Willis]

Comment on attachment 238341 [details] [diff] [review]
rev1 - updates timezones

This won't solve all our ills. Removing review request.

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

This is going to be easiest to test if we get the known timezone related bugs fixed first.  Adding dependencies on those.

Comment 10

[Matthew (lilmatt) Willis]

Questions that we need to solve with this:
- Do we recreate all the 20050125_1 zones so that they go back before 1970? If so, are we prepared to deal with the enormous VTIMEZONES that will be generated by doing so?

- Should we auto-upgrade people to use the latest version of their selected timezone? If so, should we auto-upgrade their events as well so that an event scheduled for "4 P.M." in the date period that changed stays at 4 P.M.?

- How do we want to handle removed and/or renamed timezones?

- Should we just implement a service and be done with it?

Comment 11

[Simon Paquet [:sipaq]]

I question the blocking0.3+ status here.
From what I read here, this bug presents a very high risk very late in the release cycle to questionable gain (the changes come into effect in Spring 2007). At that time, we'll probably have released 0.5 or even 0.7, so I see no reason, why this can't be postponed until after the 0.3 release.

Comment 12

[Simon Paquet [:sipaq]]

see comment 11 for why I think that this is no longer a blocker.

Comment 13

[Dan Mosedale (:dmosedale, :dmose)]

This is definitely higher risk than I realized when I originally approved it.  Sipaq's reasoning seems sound.  Minusing.

Comment 14

[Matthew (lilmatt) Willis]

Since this is no longer blocking 0.3, -> nobody@m.o

Comment 15

[Matthew (lilmatt) Willis]

Upping the priority of this.
The US daylight savings change is in March, and we need this fixed before then.

Comment 16

[Matthew (lilmatt) Willis]

Moving this to dmose as part of his timezone work.

Comment 17

[Dan Mosedale (:dmosedale, :dmose)]

Reassigning to nobody so that we can load balance better, as I haven't yet started working on it.

Comment 18

[Matthew (lilmatt) Willis]

Using the 2007a Olson timezone data, I've found the following changes.
Note that timezones which only changed their icalString (therefore only the dates of when DST starts/ends) are not listed here, as they provide no real problem once we import the database and bump the timestamp.



Renamed timezones
    "/mozilla.org/20050126_1/Africa/Asmera",
    became
    "/mozilla.org/20050126_1/Africa/Asmara",

    "/mozilla.org/20050126_1/Atlantic/Faeroe",
    became
    "/mozilla.org/20050126_1/Atlantic/Faroe",

Present in original tz but not in new
    "/mozilla.org/20050126_1/Africa/Timbuktu",
    "/mozilla.org/20050126_1/America/Argentina/ComodRivadavia",
    "/mozilla.org/20050126_1/America/Louisville",
    "/mozilla.org/20050126_1/Europe/Belfast",
    "/mozilla.org/20050126_1/Pacific/Yap",

Present in new tz but not in original
    "/mozilla.org/20050126_1/America/Atikokan",
    "/mozilla.org/20050126_1/America/Blanc-Sablon",
    "/mozilla.org/20050126_1/America/Indiana/Petersburg",
    "/mozilla.org/20050126_1/America/Indiana/Vincennes",
    "/mozilla.org/20050126_1/America/Moncton",
    "/mozilla.org/20050126_1/America/North_Dakota/New_Salem",
    "/mozilla.org/20050126_1/Australia/Currie",
    "/mozilla.org/20050126_1/Australia/Eucla",
    "/mozilla.org/20050126_1/Europe/Guernsey",
    "/mozilla.org/20050126_1/Europe/Isle_of_Man",
    "/mozilla.org/20050126_1/Europe/Jersey",
    "/mozilla.org/20050126_1/Europe/Podgorica",
    "/mozilla.org/20050126_1/Europe/Volgograd",

Only lat/long changes
    "/mozilla.org/20050126_1/America/Argentina/Tucuman",
    "/mozilla.org/20050126_1/America/Montserrat",
    "/mozilla.org/20050126_1/America/Scoresbysund",
    "/mozilla.org/20050126_1/Pacific/Easter",

Lat/long changes in addition to tz change
    "/mozilla.org/20050126_1/America/Inuvik",
    "/mozilla.org/20050126_1/America/Rainy_River",
    "/mozilla.org/20050126_1/America/Rankin_Inlet",

Comment 19

[Matthew (lilmatt) Willis]

Attachment: Updates tzdata.c and mktzdata.pl

This updates the timezones to the latest information from Olson.
Historical timezone information is included.  This may cause Outlook compatibility issues.  We'll need to test that.

It also includes a couple small (necessary) fixes from bug 368121 to add the tzidPrefix attribute.

Comment 21

[Matthew (lilmatt) Willis]

Comment on attachment 253208 [details] [diff] [review]
Updates tzdata.c and mktzdata.pl

Marking obsolete in favor of attachment #253300 [details] [diff] [review] in bug 368121.

Comment 22

[Matthew (lilmatt) Willis]

Attachment #253300 [details] [diff] checked in on SUNBIRD_0_3_BRANCH and LIGHTNING_0_3_BRANCH.

Once we smoketest the 0.3.1pre builds with this patch, we can consider landing this on MOZILLA_1_8_BRANCH and trunk.

-> [fixed0.3.1]

Comment 23

[Michiel van Leeuwen (email: mvl+moz@)]

This should have landed on the trunk too. The trunk is for baking patches. Branches are for stability. Please land on trunk.

Comment 24

[Matthew (lilmatt) Willis]

Patch checked in on MOZILLA_1_8_BRANCH and trunk.

-> FIXED

Comment 25

[cmtalbert]

VERIFIED on 0.3.1 -- Lightning: 2007020304 (Thunderbird: version 2 beta 2 (20070131)) AND on Sunbird: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.9a1) Gecko/20070203 Sunbird/0.3.1

(needs Verifying on 0.5)

Comment 26

[James Andrewartha]

Australia/Perth has DST for the next three years, with a referendum after - http://www-1.ibm.com/support/docview.wss?rs=899&uid=swg21250452

It's in the 2006p-1 and 2007a Olson DBs, dunno why it didn't get updated with the other timezones in this bug.

Comment 27

[Matthew (lilmatt) Willis]

(In reply to comment #26)
> Australia/Perth has DST for the next three years...

I opened bug 369543 for this.

Comment 28

[Omar Bajraszewski]

VERIFIED on Sunbird 0.5 RC1 Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.8.1.4pre) Gecko/20070524 Sunbird/0.5 and lightning 0.5 RC1