1503731 - Date pattern detection in datetimepicker may fail depending on the user's timezone

Status: RESOLVED FIXED

(Calendar :: Lightning Only, defect)

Description

[andycher11]

Attachment: alert.JPG

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36

Steps to reproduce:

I create a new event, set a repeat every day or every week, and get a warning that The Until date occurs before the start day, and it does not close.


Actual results:

a warning that The Until date occurs before the start day, and it does not close. So i need to close thunderbird through task manager. But if checked to repeat forever, all work good. It was in lightning 6.2.2.1 and after last update 6.2.3 still present. But in last and previous beta all work good. 


Expected results:

Creating repeating event

Comment 1

[[:MakeMyDay]]

What other addons do you have installed? Does this occur also if you have disabled all of them but Lightning?

Do you get any messages in the erreor console while reproducing the issue?

What is the default timezone you have configured in TB Options > Calendar > General > Timezone?

What is the setting in TB Options > Advanced > Date and Time Formatting?

What do you get if you are putting the following into the bar at the bottom of the error console and pressing <Enter> subsequently:

Intl.DateTimeFormat(undefined).resolvedOptions().timeZone;

Comment 2

[andycher11]

1. disabling plugins does not change anything,I left only lightning.

2.in error console :

While creating services from category 'profile-after-change', could not create service for entry 'calendar-backend-loader', contract ID 'service,@mozilla.org/calendar/backend-loader;1'
Use of Mutation Events is deprecated. Use MutationObserver instead.  calendar-widgets.xml:512:20
2018-11-01 16:39:24	gloda.NS	WARN	new proc ignoring attrib: id
Unknown property ‘user-select’.  Declaration dropped.

3.Europe/Minsk

4.Regional settings logale: Russian (Belarus)

5. Intl.DateTimeFormat(undefined).resolvedOptions().timeZone;
   "Europe/Minsk"

Comment 3

[[:MakeMyDay]]

Thanks. Can you describe in detail (where did you click, type, etc) what you exactly did until the alert was displayed?

Comment 4

[andycher11]

i add video sample. https://drive.google.com/open?id=1X3uqkgucPfk4fOVKA044HS4imOYSCs0o
and i notice another bug at this place. https://drive.google.com/open?id=1GpZuTNSkzcSNAPiaXcEF6euL2Ja2VHOc
maybe they are related.
in the second case i have in error console :


While creating services from category 'profile-after-change', could not create service for entry 'calendar-backend-loader', contract ID 'service,@mozilla.org/calendar/backend-loader;1'
Use of Mutation Events is deprecated. Use MutationObserver instead.  calendar-widgets.xml:512:20
RangeError: date value is not finite in DateTimeFormat.format() datetimepickers.xml:1875:20
	dateTimeFormatFormatToBind self-hosted:2547:12
	dateTimeFormatFormatToBind self-hosted:976:17
	formatTime chrome://calendar/content/datetimepickers/datetimepickers.xml:1875:20
	update chrome://calendar/content/datetimepickers/datetimepickers.xml:590:35
	onDatePick chrome://calendar/content/datetimepickers/datetimepickers.xml:1430:13
	onchange chrome://lightning/content/lightning-item-iframe.xul:1:1
	fireEvent chrome://calendar/content/datetimepickers/datetimepickers.xml:1494:13
	update chrome://calendar/content/datetimepickers/datetimepickers.xml:329:25
	parseTextBoxDate chrome://calendar/content/datetimepickers/datetimepickers.xml:357:13
	onchange chrome://lightning/content/lightning-item-iframe.xul:1:1
	fireEvent chrome://calendar/content/widgets/minimonth.xml:971:13
	update chrome://calendar/content/widgets/minimonth.xml:1104:17
	onPopup chrome://calendar/content/datetimepickers/datetimepickers.xml:343:13
	onpopupshowing chrome://lightning/content/lightning-item-iframe.xul:1:1

Comment 5

[andycher11]

this errors are reproduced on all three computers on which I tried (win 10 and win 8.1 with lightning 6.2.2.1 and 6.2.3)

Comment 6

[[:MakeMyDay]]

Thanks. Unfortunately, I cannot reproduce this here on Win10 with TB 60.3/Lightning 6.2.3, so it might be related to datetime-related system settings. Does that also occur if you try to set the end of the series to another day then the last day of the month?

The second issue you found is what's reported in bug 1481370 - since you haven't installed/enabled Provider for Google addon as per comment 2, that seems to be a Lightning issue, though I also cannot reproduce this behaviour.

Comment 7

[andycher11]

I made clean install of win10 on virtual machine and clean install of tb63.0 with lightning 6.2.3 and  this error is absent. I will try to compare it with three other computers, where i tried to delete tb with all profiles, reinstall it but without result.

Comment 8

[[:MakeMyDay]]

If you don't see the error in the clean installation, can you try to set the system date of the VM back to November, 1 and check whether the issue re-appears in the new profile, where it's absent now? Just to make sure that this is not a side effect of the timezone handling since we had DST transition in some regions recently (I am aware you don't have DST anymore in White Russia).

Comment 9

[andycher11]

I found this.
I have UTC +3:00 Minsk timezone in windows, when i set to UTC +3:00 Moskow the error disappears and everything works fine, even without restarting thunderbird. Trying on windows 10 1803 and 1809 with last updates.

Comment 10

[andycher11]

In clean installation I forgot to set up timezone, and it was set to Moskow. Setting time to November,1 does not result, only changing to another timezone in Windows. I also check at win 8.1 with the same result. When i set it to Minsk, i have this 2 this errors, when Moskow - everything works fine.

Comment 11

[[:MakeMyDay]]

Interesting. Can you set please set TB Options > Advanced > Date and Time Formatting to OS Regional Settings, then open the troubleshooting information (Menu > Help > Troubleshooting information) and post here what you get in the Internationalization & Localization section at the end once with OS timezone set to Minsk and once set to Moscow (you would need to close and reopen the troubleshooting page after changing the OS timezone)?

Comment 12

[andycher11]

I get the same result in both cases:

Application Settings
Requested Locales 	["en-US"]
Available Locales 	["en-US"]
App Locales 	["en-US"]
Regional Preferences 	["ru-RU"]
Default Locale 	"en-US"
Operating System
System Locales 	["ru-RU"]
Regional Preferences 	["ru-RU"]

Comment 13

[[:MakeMyDay]]

That's unexpected. Can you try that again but reboot after changing the OS timezone? It looks like the change wasn't picked up.

Comment 14

[[:MakeMyDay]]

If was still the same then, can describe how you set your timezone on Win 10?

Comment 15

[andycher11]

Attachment: timezone.jpg

Comment 16

[andycher11]

The previous post was with Windows 8.1
On windows 10 troubleshooting reports are the same too:
Application Settings
Requested Locales 	["en-US"]
Available Locales 	["en-US"]
App Locales 	["en-US"]
Regional Preferences 	["ru-BY"]
Default Locale 	"en-US"
Operating System
System Locales 	["ru-RU"]
Regional Preferences 	["ru-BY"]

Comment 17

[[:MakeMyDay]]

Thanks, on what system architecture do you observer the issues, is this 32 or 64 bit?

To get further to the culprit, can you do two additoinal checks with changing the OS timezone and checking whether the issue occurs or disappears:

1. Nairobi
2. Baku (if this fails, do the values you see in case of the second issues look different then for Minsk?)

Comment 18

[andycher11]

64 bit
1. Nairobi - whithout errors
2. Baku - without errors too

Comment 19

[[:MakeMyDay]]

For Baku, do you get as well 30.12.0002 21:55 for the second issue?

Comment 20

[andycher11]

yes, for Baku i have no errors. Only Minsk 30.12.0002 22:50 and Istanbul 30.12.0002 22:55 . They differ only 22:50 and 22:55.

Comment 21

[[:MakeMyDay]]

Attachment: {e2fda1a4-762b-4020-b5ad-a41df1933103}.xpi

The attached version of Lightning is 6.2.3 (current latest version for TB60) (en-US) with some debug code to obtain some information what arguemt is being passed when the error occurs. Can you please install that on a TB60 installation (you can overinstall it to the existing Lightning installation)?

After doing so, please

1. Go to the extensions section of the addons manager
2. Install the attached Lightning version from file (available from the little gear-wheel at the top)
3. restart TB
4. check the Extensions section at Menu > Help > Troubleshooting - Lightning should be listed with version 6.2.3tzdebugging
5. go to TB config editor (Menu > Options > Advanced > General)
6. enter calendar.debug.log into the search bar of the preference editor
7. enable that preference
8. Open the error console and clear it
9. switch to TB main window and reproduce the issue(s)
10. Post messages you get in the error console

What you should be looking for is an error message starting with (Failed to format time value...) and subsequently a log message.

To get back to your current Lightning version once we have collected enough debugging data, do follow this instructions (including uninstalling the debugging version):

https://support.mozilla.org/en-US/kb/calendar-updates-issues-thunderbird#w_lightning-disappears-after-a-thunderbird-update-release-and-beta-versions

You can cross-check at the troubleshooting page that Lightning is back to version 6.2.3

Comment 22

[andycher11]

Attachment: debconsole.JPG

i have only this message in log:

Lightning:Failed to format time value "Invalid Date" (object)  datetimepickers.xml:1878
Lightning:Invalid Date  datetimepickers.xml:1879

Comment 23

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging2

Thanks. Can you check with this version again?

Comment 24

[andycher11]

Attachment: debconsole2.JPG

now i have:
Lightning:Failed to format time value "Invalid Date" (object)  datetimepickers.xml:1938
Lightning:Invalid Date  datetimepickers.xml:1939

Comment 25

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging3

Hmm, that was unexpected. One more try if you don't mind.

Comment 26

[andycher11]

Attachment: debconsole3.JPG

Lightning:Failed to format time value "Invalid Date" (object)  datetimepickers.xml:1948
Lightning:RangeError: date value is not finite in DateTimeFormat.format()
Stack trace:
formatTime@chrome://calendar/content/datetimepickers/datetimepickers.xml:1946:24
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:626:35
onDatePick@chrome://calendar/content/datetimepickers/datetimepickers.xml:1466:13
onchange@chrome://lightning/content/lightning-item-iframe.xul:1:1
fireEvent@chrome://calendar/content/datetimepickers/datetimepickers.xml:1548:13
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:350:25
parseTextBoxDate@chrome://calendar/content/datetimepickers/datetimepickers.xml:383:13
onchange@chrome://lightning/content/lightning-item-iframe.xul:1:1
fireEvent@chrome://calendar/content/widgets/minimonth.xml:971:13
update@chrome://calendar/content/widgets/minimonth.xml:1104:17
onPopup@chrome://calendar/content/datetimepickers/datetimepickers.xml:364:13
onpopupshowing@chrome://lightning/content/lightning-item-iframe.xul:1:1
  datetimepickers.xml:1949
Lightning:Failed to format time value "Invalid Date" (object)  datetimepickers.xml:1948
Lightning:RangeError: date value is not finite in DateTimeFormat.format()
Stack trace:
formatTime@chrome://calendar/content/datetimepickers/datetimepickers.xml:1946:24
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:626:35
onDatePick@chrome://calendar/content/datetimepickers/datetimepickers.xml:1466:13
onchange@chrome://lightning/content/lightning-item-iframe.xul:1:1
fireEvent@chrome://calendar/content/datetimepickers/datetimepickers.xml:1548:13
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:350:25
parseTextBoxDate@chrome://calendar/content/datetimepickers/datetimepickers.xml:383:13
onxblblur@chrome://calendar/content/datetimepickers/datetimepickers.xml:415:15
  datetimepickers.xml:1949
Lightning:Failed to format time value "Invalid Date" (object)  datetimepickers.xml:1948
Lightning:RangeError: date value is not finite in DateTimeFormat.format()
Stack trace:
formatTime@chrome://calendar/content/datetimepickers/datetimepickers.xml:1946:24
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:626:35
onDatePick@chrome://calendar/content/datetimepickers/datetimepickers.xml:1466:13
onchange@chrome://lightning/content/lightning-item-iframe.xul:1:1
fireEvent@chrome://calendar/content/datetimepickers/datetimepickers.xml:1548:13
update@chrome://calendar/content/datetimepickers/datetimepickers.xml:350:25
parseTextBoxDate@chrome://calendar/content/datetimepickers/datetimepickers.xml:383:13
onxblblur@chrome://calendar/content/datetimepickers/datetimepickers.xml:415:15
  datetimepickers.xml:1949

Comment 27

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging4

Okay, this one will produce a lot more debug information, probably too much for a screenshot.

After installing and restarting TB, please open the error console, clear it and start to reproduce. Once done, please select all entries (right-click to the error console and click select all), copy them and paste/upload it to this bug.

Comment 28

[andycher11]

Attachment: debconsole4.txt

that's what i got

Comment 29

[andycher11]

Attachment: WithOutError_Moscow_utc3.txt

I added for example the case without errors, when the timezone is set to Moscow

Comment 30

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging5

Alright, take 5. Please post all of the messages after initial clearing the error console as described in comment 27.

Comment 31

[andycher11]

Attachment: debconsole5.txt

Comment 32

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging6

Thanks, but we're still not there. This version is even more verbose, so please clear the error console after opening the dialog before starting to reproduce.

Comment 33

[andycher11]

Attachment: debconsole6.txt

it for Minsk timezone

Comment 34

[andycher11]

Attachment: debconsole6_WithOutError_Moscow_utc3.txt

and thit for Moskow timezone whithout error, when doing the same

Comment 35

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging7

Thanks, that was helpful. This one will further narrow down to get the culprit. You'll just need to check this with Minsk time. Again, please clear the error log after opening the event dialog before reproducing.

Comment 36

[andycher11]

Attachment: debconsole7.txt

Comment 37

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging8

Obviuosly the day cannot be parsed correctly, let's see what really happens. Please start capturing the log as before.

You seem to use short date format (TB options > Calendar > General > Date Text Format) - does the issue also occur if you change to long format?

Comment 38

[andycher11]

I checked, it has already been switched to long formar, in a short format the problem is also confirmed

Comment 39

[andycher11]

Attachment: long.txt

Comment 40

[andycher11]

Attachment: short.txt

i added 2 variants

Comment 41

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging9

Ok, we're getting close to the culprit now. Please check again with debug version 9 (checking with either long or short format is enough)

Comment 42

[[:MakeMyDay]]

Attachment: Ltn 6.2.3 tzdebugging10

After checking with debug version 9 and posting the results, can you please check also with debug version 10 (that includes an attempt of a fix, but I need also the results from v9, so please provide those results first)?

Comment 43

[andycher11]

Attachment: debconsole9.txt

variant 9

Comment 44

[andycher11]

Attachment: debconsole10.txt

v10 works fine, without errors

Comment 45

[[:MakeMyDay]]

Thanks. Did that resolve both issues?

Comment 46

[andycher11]

yes, problem solved

Comment 47

[[:MakeMyDay]]

Attachment: AvoidUnexpectedTimezoneConversionInDatetimepicker-V1.diff

The root cause here is the lack of full history of our timezone definition in zones.json (compared to the built-in definition used by JS), which caused several issues since we had to change to the new datetime formatting. For Minsk and Istanbul there has been a tz change (no dst anymore) since the the probe date we use for pattern detection in the datetimepicker. 

This patch makes sure to use UTC and avoid timezone converion when doing the detection.

More general alternate approaches would be
- work around the mismatch in datetime formatter code (which would blow that code significantly)
- always use JS timezone definition in the datetime implementations (which would require to touch libical code for that, probably nothing we want to do)
- Have the full history of timezone changes in zones.json - see also [1]. However, I'm not sure if that would have performance impact.

All of that approaches might be quite regression prone, so I suggest to go with the current patch regardless of whether we follow one of the above for a more general fix.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1494160#c2

Comment 53

[andycher11]

Im sorry, but may i ask when this bug will be fixed in the prod? My users take out my brain every day ((

Comment 54

[[:MakeMyDay]]

Maybe still TB60.4 if we're quick with review and uplifts. Philipp, can you take a look at the patch?

Comment 56

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/66eedcaff9fa
Avoid timezone conversion for date pattern detection of the datetimepicker; r=philipp

Comment 57

[Jorg K (CEST = GMT+2)]

Beta (TB 65, Cal 6.7):
https://hg.mozilla.org/releases/comm-beta/rev/1f0cb7f8dc0bf68efab5f02b3b77822210b73090

Comment 58

[Jorg K (CEST = GMT+2)]

Backout from beta:
https://hg.mozilla.org/releases/comm-beta/rev/fe6b4b76613f02e23c4773e110a354ea072edbcf

It causes an abort of the test suite on Mac. Compare the beta push:
https://treeherder.mozilla.org/#/jobs?repo=comm-beta&revision=ea62abed1629265310efdd303e24186a7239dbd0

with a beta try where I've backed this out:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=1e456eb5cd3c4845aacd338e3e30c7bc302d0f80

I'm currently doing try runs on trunk to see what happens there.

Comment 59

[Jorg K (CEST = GMT+2)]

Compare with patch:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=e9e99a0ef11b6915d96f41bd52679a4c11f998d8

to patch backed out:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=55f2479b01b7f6a60d1bfbf282adfbae0b7ca3af

Comment 60

[Jorg K (CEST = GMT+2)]

Backout from trunk:
https://hg.mozilla.org/comm-central/rev/d2d18de2ec8571ec04a247b7f7096179328e4d6f
Backed out changeset 66eedcaff9fa (bug 1503731) for test failures on Mac. a=backout

Comment 61

[[:MakeMyDay]]

Philipp, could you please run the cal-recurrence/testDailyRecurrence.js mozmill test with the patch from this bug applied, so we can see why this fails on Mac (and upload a screenshot if possible)? From debugging, the first event dialog in this test doesn't close and so the test times out on automation.

Codewise, I currently see no reason why this patch should break the test.

Comment 62

[Philipp Kewisch [:Fallen][📱📆]]

Will try to get to this today

Comment 63

[Philipp Kewisch [:Fallen][📱📆]]

I've tested this and the mozmill test works with and without the patch. The event dialog closes fairly fast, so it does not look like a timing issue. Maybe another try run if this was just an intermittent?

Comment 64

[Magnus Melin [:mkmelin]]

I've sent this to try - https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=8149cecc08b8e5212196137478a213331454b013

This bug is preventing multiple xbl removals from landing, so we should make sure to fix it soon.

Comment 65

[Magnus Melin [:mkmelin]]

Not intermittent it seems.
New try with some dumps around the changed code: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=1fb2fa30cbef739ba6578fb43fcf3215bd5c350b

Comment 66

[[:MakeMyDay]]

Attachment: AvoidUnexpectedTimezoneConversionInDatetimepicker-V2.diff

This patch should fix the failing tests. In fact, it was not a test issue but a real problem.

It was not related to Mac but to the OS timezone the maschine is in. The code to init datetime patterns was mixing up month and day. As a result, dates could not be formaated correctly if the timezone offset to GMT was positive.

I changed the month and day numbers now for the probe date, so the code cannot choke on it anymore.

A try push with all my debug code in it was already green, let's see what another one with this patch will bring.

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=ede99cf6a7a9e5ee791e572461e1aba6885677d0

Comment 67

[[:MakeMyDay]]

Comment on attachment 9039432 [details] [diff] [review]
AvoidUnexpectedTimezoneConversionInDatetimepicker-V2.diff

The try push looks good. Please also grant approvals for uplifting when you're doing the re-review. It would be great if we can land this in time to include it into the next beta.

Comment 68

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 9039432 [details] [diff] [review]
AvoidUnexpectedTimezoneConversionInDatetimepicker-V2.diff

Review of attachment 9039432 [details] [diff] [review]:
-----------------------------------------------------------------

Did you want to run this on beta before we push it down to ESR? I'm fine with ESR so a+, but maybe it makes sense to test it in a version that has some more users first.

Comment 69

[[:MakeMyDay]]

Thanks. Beta would be nice if we have one before the next ESR release, since there are some other bugs around which i hope are getting fixed with this change, but that needs to be confirmed by the reporters.

If there's no new beta we can go directly to ESR.

Comment 70

[Jorg K (CEST = GMT+2)]

TB 66 Beta 66 planned as soon as one blocker bug is resolved. TB 60.6 is planned for end of March unless something terrible happens.

Comment 71

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5fcb177a3309
Avoid timezone conversion for date pattern detection of the dettimepicker;r=philipp

Comment 72

[Richard Marti (:Paenglab)]

Philipp, version 6.9 is missing as target.

Comment 73

[Jorg K (CEST = GMT+2)]

TB 66 beta 3 (if we do one) / Cal 6.8:
https://hg.mozilla.org/releases/comm-beta/rev/68eeb51034c66fa5ae08c2ed0975e77bb8a5e2c6

Due to the prior landing and later backout I didn't notice that this needed re-landing :-(

Comment 74

[Jorg K (CEST = GMT+2)]

TB 60.6 ESR / Cal 6.2.6:
https://hg.mozilla.org/releases/comm-esr60/rev/20a503e373535088938fab27375c8c547d078c56