937261 - Wrong time calculation - regression

Status: RESOLVED DUPLICATE of bug 1346211

(Core :: JavaScript Engine, defect)

Description

[Gustavo Homem]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20130911160237

Steps to reproduce:

1. login to Zimbra
2. click on the calendar tab
3. right click on a free time slot
4. choose "New Appointement"
5. give the Appointement a title
6. click OK


Actual results:

There is a transient window on the top of the calendar saying "Appointement Created", as expected, but the appointement does NOT get created. The appointment does not appear even after logout / login, so it is not a visulization problem.


Expected results:

The appointment should have been created and the calendar view should have been refreshed.

Comment 1

[Gustavo Homem]

This is reproducible on the Zimbra demo:

http://www.zimbra.com/products/hosted_demo.php

This worked without problems on earlier versions of Firefox (17 ESR works, some of the releases between 17 and 24 probably worked - can't be 100% sure which release made it stop working).

Comment 2

[Gustavo Homem]

Update: works on FF25 on Win7, Fails on FF25 on Ubuntu. Works on Chrome 29.0.1547.76 on Ubuntu.

Comment 3

[Bogdan Maris, Desktop Test Engineering]

It works for me on Ubuntu 12.04 x32, Windows 7 x64 and Mac OS X 10.9 using Firefox 25.0.1 (buildID: 20131112160018).

Comment 4

[Gustavo Homem]

Bogdan Maris,

How did you install Firefox on Ubuntu? I need to narrow the problem. 25.01 is not on the repositories yet, it seems.

Comment 5

[Bogdan Maris, Desktop Test Engineering]

Here you can find Firefox 25.0.1.
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/25.0.1-candidates/build1/linux-i686/en-US/

Comment 6

[Gustavo Homem]

Bogdan,

I just tested exactly the binary of Firefox binary 25.01 you posted, in two Ubuntu 12.04 32bits computers at it doesn't work in any of them.

It fails on three installations of Zimbra, one of which is the demo I linked on this report. The appointement never appears in the calendar although the transient "appointment created" window briefly appears.

Comment 7

[Gustavo Homem]

We made a clean install of Ubuntu 12.04, after which Firefox version is 11 and appointments work. Then after

apt-get update
apt-get install firefox firefox-globalmenu firefox-gnome-support firefox-locale-en firefox-locale-pt

we get Firefox 25 and creating appointments doesn't work anymore.

To summarize, we reproduced this in all machines where we tried, so far:

- plain Ubuntu 12.04 install, with Unity, and Firefox 25 from the repos (2x)
- plain Kubuntu 12.04 install and Firefox 25 from the repos
- custom Ubuntu 12.04 KDE based install and Firefox 25 from the repos (tested on 4 machines)
- custom Ubuntu 12.04 KDE based install and Firefox 25 from the nightly build binaries (via Bogdman Maris)

We had 3 people testing to avoid any kind of misunderstanding.

I updated the status to UNCONFIRMED, since it seems that there is really a problem.

Comment 8

[Gustavo Homem]

After all the events are created but with one month of offset. I just found out that December was populated with all my failed creation trials during November. 

Plus, bug affects only the month of November as on October and January I can create events without any problem. At least for November 2012 and 2013 the off-by-one-month problem is reproducible. I will attach a screenshot.

Comment 9

[Gustavo Homem]

Attachment: Screenshot of the problem during November.

Comment 10

[[:Aleksej]]

Please make sure you have actually tested with a Firefox from Mozilla, as the Ubuntu one can differ, and if a Firefox is running and you don't specify a profile or -no-remote / MOZ_NO_REMOTE=1, the new Firefox you start will just create a new window of the running one.

You can get other releases from https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

There are also nightly builds at https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ , which you can bisect using mozregression http://harthur.github.io/mozregression/  The release dates are not the same as those on the Nightly branch, see version numbers in the file names.

Comment 11

[[:Aleksej]]

There is bug 937575 for Gmail / Google Contacts showing December instead of November.

Comment 12

[Gustavo Homem]

Aleksej,

This is just as with bug 937575. 

Fails for Feb, Apr, Jun, Sep, Nov.

I did test Firefox 25.0.1. from mozilla and ran it without any other browser running (I am aware of the -no-remote vs browser reuse thing).

Comment 13

[Gustavo Homem]

Still a problem with Firefox 27 on Ubuntu and Zimbra 8.0.6_GA_5922 (build 20131203103705). Would be great to know if the bug is on Firefox or Zimbra.

Comment 14

[Gustavo Homem]

News on this: I used the bissection tool.

---------------------------

mozregression --good=2012-12-01
[...]
Got as far as we can go bisecting nightlies...
Ensuring we have enough metadata to get a pushlog...
Last good revision: 8592c41069c2 (2013-01-11)
First bad revision: 1761f4a9081c (2013-01-12)
Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8592c41069c2&tochange=1761f4a9081c

... attempting to bisect inbound builds (starting from previous day, to make sure no inbound revision is missed)
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/01/2013-01-10-03-09-39-mozilla-central/firefox-21.0a1.en-US.linux-i686.tar.bz2
===== Downloaded 100% =====
Installing nightly
Getting inbound builds between 0a6e5a67c4e8 and 1761f4a9081c
Oh noes, no (more) inbound revisions :(
do you want to bisect further by fetching the repository and building? (y or n) n

---------------------------

So something got broke around here:

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8592c41069c2&tochange=1761f4a9081c

Comment 15

[Gustavo Homem]

Suspect?

https://bugzilla.mozilla.org/show_bug.cgi?id=937261

Comment 16

[[:Aleksej]]

(https://hg.mozilla.org/mozilla-central/rev/5a88ed3a3ace)

Comment 17

[Andrea Marchesini [:baku]]

I think this bug is already fixed by: bug 830257. Waldo, any thought?

Comment 18

[Gustavo Homem]

If it is fixed, is there a nightly to test it? 

The bug is present on Firefox 27.

Comment 19

[Gustavo Homem]

@Andrea the bug is still present on Firefox 28.

Comment 20

[Gustavo Homem]

This bug is still present on Firefox 33. Since we know when this got broken can you have a comment an when it will be fixed?

Comment 21

[Gustavo Homem]

Additional tests reveals that this depends on the timezone. Summary of testing setting TZ from the shell:

Working:
 TZ=US/Pacific firefox
 TZ=UTC-1 firefox
 TZ=UTC+1 firefox
 TZ=Europe/Madrid firefox
 TZ=Atlantic/Azores firefox
 
Broken:
 TZ=Europe/Lisbon firefox
 TZ=Portugal firefox

The problem can be reproduced by running, for example:

  TZ=Europe/Lisbon firefox

Comment 22

[Gustavo Homem]

Additional info:

TZ=Europe/London firefox

works. But 

TZ=Europe/Lisbon firefox

does NOT.

It's the same time including daylight savings.

Comment 23

[Gustavo Homem]

Still a problem in Firefox 3.5. Can someone change the status to confirmed?

You only need to run

TZ=Europe/Lisbon firefox

an schedule an event in February using this Zimbra server:

http://www.zimbra.com/try/zimbra-collaboration-hosted-demo

I can arrange a temporary mailbox on a Zimbra server if that helps.

Comment 25

[Andrea Marchesini [:baku]]

Can you reproduce this issue? If not, can you close this bug? Thanks!

Comment 26

[Andrea Marchesini [:baku]]

I close this bug. In case someone is able to reproduce this issue, please, open it again and NI me.

Comment 27

[nick-mozillabugzilla]

Doing the test at https://bugzilla.mozilla.org/show_bug.cgi?id=937575#c0 with Iceweasel (Firefox) 38.2.0, I do find that this now also works for me.

With some previous FF version(s) (and I don't remember when), that test failed for me.

Comment 28

[Gustavo Homem]

The bug is reproducible now:

1- run

TZ=Europe/Lisbon firefox

2. Go to a Zimbra webmail interface
3. Create an event in November
4. Check that the event is created one month later

Mailed Andrea Marchesini additional info and testing means.

Firefox 42 on Linux.

Comment 29

[Andrea Marchesini [:baku]]

Here the issue, your code does:

var a=new Date(0,0,1,0,0,0,0);
if(o.year!=null){
a.setFullYear(o.year) // 2015
}
if(o.month!=null){
a.setMonth(o.month) // 10 in my tests (10 is november)
}
...

At this point the date is: Tue Dec 01 2015 23:00:00 GMT+0000 (WET)
because november doesn't have 31 days. If you set the day before the month it works.

Waldo, is this the correct behavior of setMonth?

Comment 30

[Gustavo Homem]

Great catch Andrea!

I can confirm the problem on Firefox's web console:

TZ=Europe/Lisbon firefox

a=new Date(0,0,1,0,0,0,0);
a.setFullYear(2015);
a.setMonth(10);
a.getMonth();

--> returns 11 (WRONG)

TZ=Europe/London firefox

a=new Date(0,0,1,0,0,0,0);
a.setFullYear(2015);
a.setMonth(10);
a.getMonth();

--> returns 10 (CORRECT)

Comment 31

[Andrea Marchesini [:baku]]

Good! I close this bug. Thanks for helping me to debug it.

Comment 32

[Gustavo Homem]

Don't close this bug! 

The bug is a bug as you can see from what I posted above and was introduced as described here:

https://bugzilla.mozilla.org/show_bug.cgi?id=937261#c14

This needs further investigation on why Javascript is returning the wrong month for timezones which are equivalent.

Comment 33

[André Bargull [:anba]]

Firefox/SpiderMonkey doesn't use tzdata [1] to retrieve time zone information, instead the underlying OS is queried. Unfortunately some operating systems don't return valid results for dates before 1 January 1970 [2] which led to the "equivalent years for daylight saving" workaround in [3]. So what's happening here is that OS limitations and workarounds for said limitations produce a bogus date. (Also insert here the usual rants about JavaScript's antiquated Date API. ;-) )

[1] https://en.wikipedia.org/wiki/Tzdata
[2] https://en.wikipedia.org/wiki/Unix_time
[3] https://hg.mozilla.org/mozilla-central/file/tip/js/src/jsdate.cpp#l385


// Year parameters between 0 and 99 are relative to 1900, so the actual
// year for `a` is 1900. 1900 is before Jan. 1 1970, the equivalent year
// for 1900 is used to compute the daylight saving offset.
// => EquivalentYearForDST(1900) from jsdate.cpp returns 1973.
// Now things are getting interesting:
// Europe/Lisbon was +1:00 UTC in 1973 and did not observe DST. This one
// hour offset from UTC shifts the date from Jan 1 to Dec 31.
a = new Date(0,0,1,0,0,0,0);
console.log(a.toString()); // "Sun Dec 31 1899 23:00:00 GMT+0000 (LMT)"

// The year component gets updated to 2015, but the other components
// (month, day, hour) are still unchanged.
a.setFullYear(2015);
console.log(a.toString()); // "Thu Dec 31 2015 23:00:00 GMT+0000 (WET)"

// The month component is changed to November, but the day component is
// still 31. Since Nov 31 doesn't represent a valid date, the extra
// day after Nov 30 shifts the date to Dec 1.
a.setMonth(10);
console.log(a.toString()); // "Tue Dec 01 2015 23:00:00 GMT+0000 (WET)"

Comment 34

[Gustavo Homem]

@André,

Very interesting input from your side!

Here an explicit summary for the record.

TZ=Europe/London firefox

a = new Date(0,0,1,0,0,0,0);
console.log(a.toString()); 
a.setFullYear(2015);
console.log(a.toString());
a.setMonth(10);
console.log(a.toString()); 

RESULT (CORRECT):
Mon Jan 01 1900 00:00:00 GMT+0000 (GMT)
Thu Jan 01 2015 00:00:00 GMT+0000 (GMT)
Sun Nov 01 2015 00:00:00 GMT+0000 (GMT)

TZ=Europe/Lisbon firefox

a = new Date(0,0,1,0,0,0,0);
console.log(a.toString()); 
a.setFullYear(2015);
console.log(a.toString());
a.setMonth(10);
console.log(a.toString());

RESULT (WRONG):
Sun Dec 31 1899 23:00:00 GMT+0000 (LMT)
Thu Dec 31 2015 23:00:00 GMT+0000 (WET)
Tue Dec 01 2015 23:00:00 GMT+0000 (WET)

So, to recap:

1- this used to work well on Firefox and got broken (bissection on #c14)
2- is this setMonth behaviour correct? (what does the standard say, what do other browsers do, etc)
3- if this is an OS problem can this be reproduced a at lower OS level so that we file a bug on the respective OS package?
4- is Zimbra using something other than Javascript best practices here?

We need a couple of answers in order to understand where is more logical to fix this.

Short comments:

2- this setMonth behaviour seems strange, I understand that a.toString could fail on invalid dates and/or that an isvalid() method could make sense; I'm not sure if silently "fixing" an invalid date is the best thing. Why do we fix it to the next day instead of throwing an error or ignoring the problem?

3-

TZ=Europe/Lisbon date -d 19001130 --> works fine
TZ=Europe/Lisbon date -d 19001131 --> fails with an error

The date command doesn't allow to construct a date step by step but fails with an error (ie, not silently) if the given date is invalid.

4- will be hard to lobby for a change in Zimbra since all browsers, including Firefox, work fine on Windows and Google Chrome works fine on Linux, as did Firefox in the past. Setting the on the date before the month also has potential for getting into invalid dates if we set 31 and then set a month which is less then 31 days. It could solve this problem but would not solve the general problem of temporarily invalid dates being silently "corrected"

What to do next?

Comment 35

[André Bargull [:anba]]

(In reply to Gustavo Homem from comment #34)
> 1- this used to work well on Firefox and got broken (bissection on #c14)

It probably broke when Firefox started to use more accurate information for historical time zone changes.


> 2- is this setMonth behaviour correct? (what does the standard say, what do other browsers do, etc)

Yes, the setMonth behaviour is correct and follows the specification. It's also consistent with other browsers. 


> 3- if this is an OS problem can this be reproduced a at lower OS level so that we file a bug on the respective OS package?

No, this is not an OS problem. It works on Windows because Windows doesn't have a separate Europe/Lisbon time zone, instead it uses the same rules for Europe/Lisbon and Europe/London [1].

[1] https://msdn.microsoft.com/en-us/library/ms912391%28v=winembedded.11%29.aspx


> 4- is Zimbra using something other than Javascript best practices here?

I'd suggest to avoid using the various Date.prototype.setXXX modifier methods and instead treat Date objects as immutable objects. The code in comment 29 should be rewritten to:

```
// Or any other suitable default year.
var year = o.year != null ? o.year : 2000;
var month = o.month != null ? o.month : 0;
var day = o.day != null ? o.day : 1;
var date = new Date(year, month, day);
```

This approach avoids the pitfalls described here, but I don't know if it should be considered a known best practice. 


> 2- this setMonth behaviour seems strange, I understand that a.toString could fail on invalid dates and/or that an isvalid() method could make sense; I'm not sure if silently "fixing" an invalid date is the best thing. Why do we fix it to the next day instead of throwing an error or ignoring the problem?

The JavaScript Date API is awful. It was "borrowed" from Java, but while Java got two new APIs (java.util.Calendar and java.time in Java 8), JavaScript never received any updates. Unfortunately the current behaviour cannot be changed because other applications rely on this more lenient, error-recovering mode. 


> 4- will be hard to lobby for a change in Zimbra since all browsers, including Firefox, work fine on Windows and Google Chrome works fine on Linux, as did Firefox in the past.

It works on Windows because Windows doesn't actually support Europe/Lisbon (see above). Chrome uses a different "equivalent year" range [2] which avoids the year 1973 issue. The same applies to Safari [3].

[2] https://github.com/v8/v8/blob/35452afafbb2da49b9c6f946f42e1ef2b735bf03/src/date.h#L177
[3] https://github.com/WebKit/webkit/blob/481b86204d1e5661d6dafd91b57b89ab43c23557/Source/WTF/wtf/DateMath.cpp#L340

Comment 36

[Gustavo Homem]

Thanks a lot again for all the precious input. Things are much more clear now.

Not trying to contradict any of your findings I would say that from a practical point of view a stronger statement from Mozilla on what to do next would be useful. To put it short I see these options:

1- claim that setting dates in steps in unsupported and write it down on some knowledge base online doc (being broken on a certain timezone without willingness to fix is more than enough to de-support)
2- issue warning  or error any time a date is is set in steps(therefore pushing the web app vendors to fix on their side)
3- revert the commit that broke this (5a88ed3a3ace  ?)
4- try to fix the problem by: aligning the behaviour with chrome or safari (other reference years? which side effects?)
5- try to fix the problem by preventing setMonth from outsmarting the programmer (ie allowing invalid dates)

I am kindly asking for this because from a practical point of view I see it as unlikely that Zimbra will change code that works well on most browsers unless there is a clear statement on such how such code breaks best practices (1 and/ or 2) and is unsupported. We can't de-support Europe/Lisbon (like Microsoft does according to your comment) because all Linux distributions specifically support it and expose such configuration on the installer.

Can we find a way out of this?

Comment 37

[André Bargull [:anba]]

(In reply to Gustavo Homem from comment #36)
> Not trying to contradict any of your findings I would say that from a
> practical point of view a stronger statement from Mozilla on what to do next
> would be useful.

AFAIK Mozilla is aware that there are multiple time zone related issues in Firefox/SpiderMonkey. (Note: I'm not a Mozilla employee.)


> 1- claim that setting dates in steps in unsupported and write it down on
> some knowledge base online doc (being broken on a certain timezone without
> willingness to fix is more than enough to de-support)

The real culprit is calling `new Date(0, 0, 1, 0, 0, 0, 0)`, everything else is a consequential error. For example if `new Date(0, 0, 1, 0, 0, 0, 0)` is replaced with `new Date(2000, 0, 1, 0, 0, 0, 0)`, the error goes away.


> 2- issue warning  or error any time a date is is set in steps(therefore
> pushing the web app vendors to fix on their side)

I don't think this is possible without breaking the web. 


> 3- revert the commit that broke this (5a88ed3a3ace  ?)
> 4- try to fix the problem by: aligning the behaviour with chrome or safari
> (other reference years? which side effects?)

Probably doable, although I'd prefer Firefox would directly use tzdata as recommended by the ECMAScript specification [1, 2]. (Except it's probably not as simple as "just using" tzdata, because tzdata also needs to be updated regularly to avoid using stale time zone information, etc.)

[1] http://ecma-international.org/ecma-262/6.0/#sec-local-time-zone-adjustment
[2] http://ecma-international.org/ecma-262/6.0/#sec-daylight-saving-time-adjustment


> 5- try to fix the problem by preventing setMonth from outsmarting the
> programmer (ie allowing invalid dates)

This is also not possible without breaking other applications.


FWIW, only the following three time zones are affected by this issue:
Africa/Bissau
Africa/El_Aaiun
Europe/Lisbon

Comment 38

[Gustavo Homem]

> 
> > 1- claim that setting dates in steps in unsupported and write it down on
> > some knowledge base online doc (being broken on a certain timezone without
> > willingness to fix is more than enough to de-support)
> 
> The real culprit is calling `new Date(0, 0, 1, 0, 0, 0, 0)`, everything else
> is a consequential error. For example if `new Date(0, 0, 1, 0, 0, 0, 0)` is
> replaced with `new Date(2000, 0, 1, 0, 0, 0, 0)`, the error goes away.

Great! Now for people not to use new Date(0, 0, 1, 0, 0, 0, 0) there should be something written somewhere advising against such use. An official document.

> 
> 
> > 2- issue warning  or error any time a date is is set in steps(therefore
> > pushing the web app vendors to fix on their side)
> 
> I don't think this is possible without breaking the web. 
> 

Firefox "broke the web" several times while reinforcing security requirements of several sorts. It broke from legacy unmaintained websites to a major high profile web banking sites, from what I remember. That was done because that it the right thing to do and sites had to adapt. Besides that, it broke my timezone while trying to fix something for other timezone. 

> 
> > 3- revert the commit that broke this (5a88ed3a3ace  ?)
> > 4- try to fix the problem by: aligning the behaviour with chrome or safari
> > (other reference years? which side effects?)
> 
> Probably doable, although I'd prefer Firefox would directly use tzdata as
> recommended by the ECMAScript specification [1, 2]. (Except it's probably
> not as simple as "just using" tzdata, because tzdata also needs to be
> updated regularly to avoid using stale time zone information, etc.)
> 
> [1]
> http://ecma-international.org/ecma-262/6.0/#sec-local-time-zone-adjustment
> [2]
> http://ecma-international.org/ecma-262/6.0/#sec-daylight-saving-time-
> adjustment
> 

tzdata updates are part of the OS updates. I think it is fine to rely on those if this is the way to go.

> 
> > 5- try to fix the problem by preventing setMonth from outsmarting the
> > programmer (ie allowing invalid dates)
> 
> This is also not possible without breaking other applications.
> 

That would only break apps that rely on that "smart" behaviour from setMonth. Right now apps that rely on a standard function behaviour (accept the value or throw an error) are broken.

Anyway, would be nice if someone from Mozilla could examine this situation and suggest a way out.

Comment 39

[André Bargull [:anba]]

Dup'ing forward to bug 1346211 which should have fixed this problem. Please reopen if the issue is still present. Thanks!