1789437 - an event not displayed, but listed when searching for it ("Same occurrence found twice, protecting you from death by recursion")

Status: RESOLVED FIXED

(Calendar :: ICAL.js Integration, defect)

Description

[Henri D/CH]

Attachment: Thunderbird_calendar_bug.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0

Steps to reproduce:

URGENT, CRITICAL problem with CALENDAR in Thunderbird 102.2.1 (×64): during some calendar periods, many meetings of one of my 4 calendars are missing on the display though they indeed do exist (happens in all views from day to month). The search function does find them, but they remain invisible on the corresponding day.

Actual results:

When I change the meeting's owner, the meeting appears on the new owner calendar; when I change the owner back to the correct person, it flahes up for ¼second but then immediately disappears again. When I create a new meeting, it also flashes up for ¼second then disappears - but I still can find and modify it with the search function.
Example: see enclosed - the Gemelli-meeting of 2022-09-29 is invisible on that day but is found by the search function !!!

Expected results:

All meetings should always be displayed without any exception.

Comment 1

[Magnus Melin [:mkmelin]]

Is the event in the calendar x which you have set to hidden (click the eye)?

Comment 2

[Henri D/CH]

Attachment: Invisible_event.png

Comment 3

[Henri D/CH]

My calendar is a 108KB Henri.ics file on a web server accessed in real time by several devices and of course enabled for display (I don't know any possibility of hiding single events, only whole calendars). All events are displayed correctly on my smartphone, on my wife's iPad as well as on my laptop still running Thunderbird 91.13.0×64. My calendar is enabled and displayed in blue. One can see on the screenshot an event on the 26th but nothing on the 29th. All newly created events flash up shortly but now disappear after a fraction of seconds (independently of their date). Almost all my events in June/July/September (several dozens, surprisingly not August) have disappeared from the screen though they can be found by a search. Thunderbird has become totally useless !!!

Note that it already happened a very few times in the past years that events created on my smartphone didn't appear on Thunderbird; I blamed my Android apps (CalenGoo/CalendarSync), but it now seems more likely that some bug already was present in elder Thunderbird releases though it usually didn't create problems. Maybe there is not enough space allocated for display (?).

I will now go back to v. 91.13.0; if this doesn't work I will post here a new comment and switch to MS Outlook.

Comment 4

[Henri D/CH]

I could not switch back to v. 92.13.0 because the profile has become incompatible. When I reinstalled v. 102.2.1, about 2-3 seconds after start all invisible events flashed very shortly for a fraction of seconds and disappeared again.
Further trials:
I exported my calendar and reimported it into a new local calendar. The invisible events were in part not the same as on the online calendar.

Thus I will use another software for my calendar until the bug is resolved.

Comment 5

[Magnus Melin [:mkmelin]]

Is there anything in the error console (Ctrl+Shift+J) ?
You can also try setting the pref calendar.icaljs to false and see if it makes any difference.

Comment 6

[Henri D/CH]

Attachment: TB_error_log.txt

Comment 7

[Henri D/CH]

Attachment: TB_error_log_new.txt

Comment 8

[Henri D/CH]

The error console indeed has some contents (after deleting everything and starting TB again) - see uploaded text file.

YES [and many thanks ! ]: setting calendar.icaljs to false appears to resolve the problem, though I just checked the current September, not everything yet. However, there are still new errors in the error console after this setting change - see next uploaded text file 'new'.

By the way: Outlook displays all events correctly, too; but it is sadly impossible to modify them or create new ones on an internet .ics calendar — and by the way, I experienced soooooo many heavy bugs in MS Outlook during my last 20 years professional life that I would like to avoid it by all means! But I could use it exceptionally to check whether TB displays everything…

Comment 9

[Henri D/CH]

PS: additional problem:
After a while (1-2 hours hibernation), all events disappeared and there appeared a button "activate" on my 4 calendars; however, these buttons didn't work until I set calendar.icaljs to true again then further back to false after the calendars were activated. Needs to be followed up whether the configuration works and the calendars remain activated without calendar.icaljs having to be changed each time.

Comment 10

[Magnus Melin [:mkmelin]]

I guess this is the relevant part:

Exception { name: "NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS", message: "[JavaScript Error: "Same occurrence found twice, protecting you from death by recursion" {file: "resource:///modules/calendar/Ical.jsm" line: 7300}]

https://searchfox.org/comm-central/rev/e5d73c00f79339e837c306016f593c45a52cbd8b/calendar/base/modules/Ical.jsm#7304

Comment 11

[Magnus Melin [:mkmelin]]

I don't suppose you have an ics file you can provide as a test case?

Comment 12

[Henri D/CH]

Now I updated also my Laptop to TB 102 and got exactly the same bug with the same calendar; setting calendar.icaljs to false resolved the problem here, too. Likely my second desktop (in a different location) will behave similar.

I understand that some event is found twice, but this should of course not lead to invisible other events…
Althought I didn't see any double event, there is indeed a double UID 8309e05d-27ff-4a86-90de-fbf127a51d02 (no idea how this happened). Surprisingly, the double event's creation date and details are different!

I send you this problematic .ics calendar file separately by email.

Comment 13

[Henri D/CH]

PS: Deleting the tasks and both similar events having the same UID, then recreating this recurring event again did NOT help, the update to TB 102.2.2 neither so long calendar.icaljs is True (default). But it still seems to work correctly when calendar.icaljs is set to False.

Comment 15

[Bryan]

I created bug 1791894 and I see it's now officially marked as a dupe of this one. My original bug was marked with a severity of S3 because I was seemingly the only reporter. Wondering why this bug has no severity (nor priority) set yet? New to your flow so not sure what standard operating procedure is. I'm trying to decide if I have to ditch TB entirely or if I can limp along waiting on a fix. I REALLY don't want to have to ditch TB :-) Thanks all - your efforts are appreciated.

Comment 16

[Henri D/CH]

Happy to see that I'm not the only victim of this very heavy bug which still subsists in version 102.3.1 (×64) !
I guess that lots of people experience the problem without realizing that events are missing — and then just miss the events … It is VITAL for every user that no event could ever be missing on the display. Without the hint on calendar.icaljs (many thanks again), I would already have deinstalled Thunderbird forever.
The bug occurs on 3 different Win10 PC's of different brands/hardwares and with different software configurations - however, the missing events are strangely only partially the same on the 3 PCs though they are all connected to the same .ics calendar from the web. To avoid this highly sad problem, it is mandatory to set calendar.icaljs to false - but this is of course no sound solution for everybody and may have negative consequences on other features.
Without a bugfix or the calendar.icaljs change to False, Thunderbird is ENTIRELY USELESS : one cannot rely at all on all events to be displayed !!!
Note that I fixed my calendar: they are no double events anymore; this earlier problem is clearly not the source of the problem. Maybe a hint: when I change calendar.icaljs from True to False or conversely, upon restarting Thunderbird the missing events sometimes appear for a fraction of a second before immediately disappearing again.

Comment 17

[Geoff Lankow (:darktrojan)]

Looking at the test case, you have an event (starts 2022-01-31) that recurs on the 31st of the month and the last day of the month. Make it just the last day of the month and the problem goes away.

Comment 18

[Henri D/CH]

Many thanks, this works indeed. No idea how the error came into this event (maybe it was very exceptionally created on my Android smartphone).
But there is anyway a bug in the icaljs: obviously JS tries to hide the double occurrences of this event, but the most missing events were in September, which has only 30 days - that is, a single occurrence of this problematic event. Not finding the double occurrence, the script obviously did hide a lot of other September events instead…
There were also few events missing in July, but I don't remember to have encountered any problem in January, March and May !

Comment 19

[Bryan]

(In reply to Geoff Lankow (:darktrojan) from comment #17)

Looking at the test case, you have an event (starts 2022-01-31) that recurs on the 31st of the month and the last day of the month. Make it just the last day of the month and the problem goes away.

I can confirm that I absolutely do not have the described data condition and yet I still see the error - the error stands - even with non-repeating events.

Comment 20

[Geoff Lankow (:darktrojan)]

I'm not claiming there isn't a bug, just pointing out what triggered it in the reporter's data.

Comment 21

[Bryan]

(In reply to Geoff Lankow (:darktrojan) from comment #20)

I'm not claiming there isn't a bug, just pointing out what triggered it in the reporter's data.

Thanks Geoff - just making sure - lousy lousy very disruptive bug and I'm hoping the direction stays forward :-) Hoping against hope for progress soon....also looking for TB replacement because I'm a complete cynic and realist :-)

Cheers

Comment 22

[Bryan]

UPDATE: In my original bug report (1791894) I mentioned events laid down from a meeting invite appeared unaffected and worked as they should. Nope - now proven false.

Comment 23

[Bryan]

Just checking back to see if/when this will be assigned a priority or otherwise to see if there s any kind of an ETA on resolution? Thanks in advance for any insight into this black box experience of Bugzilla for Mozilla bugs :-)

Comment 25

[Bryan]

I'm not so sure this is a duplicate of bug 1784889 (will read more thoroughly and compare to what I have been seeing), but now that you have indicated Magnus that this is a duplicate, what does that mean? Has this been resolved in any of what appear to be a mounting number of duplicate bug reports - meaning it is indeed widespread and effecting more than just a few folks?

All I know for sure is I am now on TB ver 102.4.2 and I still have the issues I reported under bug 1791894. So if this is assumed in anyway as fixed - it does not appear that's the case.

Thanks Magnus

Comment 26

[Magnus Melin [:mkmelin]]

This bug is open (status NEW), and not yet resolved. We mark bugs duplicate if we get more than one report.
Unclear how widespread the problem is, but it exists and it would be good to fix.

Comment 27

[Bryan]

OK...went and read bug 1784889 more thoroughly - it's about the use of remote calendars and my originally reported bug 1791894 is about using the local calendar. So the workaround posted by lunic in bug 1784889 does not apply for the use of local calendars.

This is not a duplicate of bug 1784889.

The issue in my bug was recursion - Rainers debug output in bug 1784889 is clearly different than what I and Henri (submitter of the issue this is in) posted.

Thanks for reassessing this one

Comment 28

[Magnus Melin [:mkmelin]]

The error would show the same no matter if remote or local calendar, as the bug is in the parsing of the calendar data.

Comment 29

[Bryan]

Fair enough - I'm referring to the workaround suggested - makes one think that maybe remote calendars is the issue and with no dissenting opinion....here I am wondering who thinks what. Mostly frustrated with TB and this hugely brutal bug that seems to be getting nowhere fast. Thanks again - here's hoping for some traction on this sucker soon :-)

Comment 30

[Rainer]

Rainer here. I reported bug 1789437.
Yes, this is the same problem, absolutely.
Firstly, it also shows up with local calendars, see this part in my other bug report:

When I export the Calendar as an ics file through the Kolab Now web interface and import it into a local Thunderbird calendar, some of the missing events show up, but others are still not diplayed (but can again still be found in the search function).
Secondly, I also see this strange effect that some events show up for a second or (especially after fresh syncs) even longer. And then a moment later they are gone again.
I share your frustration. I use Thunderbird for business purposes, and my business needs an absolutely reliable calendar. I will have to switch to MS Outlook (or maybe eM Client), at least until this bug is resolved.

Comment 31

[Rainer]

Sorry, bad formatting above.
And I meant I reported bug 1784889.

Comment 32

[Henri D/CH]

Here again my above info summarized:
setting calendar.icaljs to false (suggestion of Magnus Melin [:mkmelin] ) appears to resolve the problem !

The bug appeared again when setting calendar.icaljs back to true, until the real cause was eliminated:
an event that erroneously recurred on the 31st of the month AND on the last day of the month.
Obviously icaljs doesn't manage this correctly and hides a lot of other events on months where the last day is not the 31th…
but it could be that they are further bugs in this routine, too.

Since the correction of the causal event, the problem didn't reappear; thus, I will not follow this bug anymore.

Comment 33

[Wayne Mery (:wsmwk)]

bug 1792561 isn't a duplicate?

Comment 34

[Rainer]

Thank you Magnus and Henri, this helped me to find my own culprit.
Here is what I did:

1. I exported all my (roughly 3000) calendar events into an ICS file.
2. I opened it with a text editor and searched for an occurence of ",-1" as this indicates "last day" (e.g. of the month) according to ICS specs (see https://icalendar.org/iCalendar-RFC-5545/3-8-5-3-recurrence-rule.html).
3. I had one hit, a calendar event with the following Recurrence Rule:
   RRULE:FREQ=MONTHLY;INTERVAL=1;BYMONTHDAY=31,-1
   So again: An event that occurs both on the 31st and the last day of the month.
4. I deleted the event in the calendar. (It already looked a bit odd since Thunderbird did not allow me to edit it, only delete it.)

It seems the problem is gone (for now). Let's see if it really works long-term.

Bottom-line: It seems there is a workaround, but please, please, please fix this horrible bug. It has cost me hours of troubleshooting, difficult conversations with clients and lots of faith in Thunderbird.

Comment 35

[Bryan]

Well hopefully now that users have done the legwork for Mozilla it can be solved. :-)

Thanks Rainer!! I had just exported to ICS yesterday to start using Evolution for my calendar...so already had the file around to check - found one instance of BYMONTHDAY=30,-1 - deleted - now seem to see my missing events back. As you said - time will tell.

I would like to 2nd Rainer's plea Mozilla folks - love Mozilla - this shook that love a whole lot

Cheers

Comment 37

[Geoff Lankow (:darktrojan)]

I have been working on this. I thought I'd updated Bugzilla but apparently not.

Comment 38

[Bryan]

LOL....same problem I have with my staff here - mark issue as "in progress" as you start the work....not when you are ready for the fix to be tested :-) Thanks Geoff!!

Comment 39

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1789437 - Remove ical.js "death by recursion" protection. r=#thunderbird-reviewers

I'm not convinced there are any situations where death by recursion is actually possible at this point. The original author of this line wasn't either.

What it does do is stop enumerating occurrences where one is the same as the previous.
There are valid reasons this might happen (such as recurrence on the last and 31st days of the month) and to stop here is as bad as death by recursion.

Depends on D161971

Comment 40

[Geoff Lankow (:darktrojan)]

Looking at this code has raised some questions about it that I'm trying to deal with upstream, but for now removing this block of code means ical.js has the same behaviour as libical did, AFAICT. So to avoid further annoyance for users I'm proposing to fix this now and come back to the other issues separately.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/975269a48251
Remove ical.js "death by recursion" protection. r=mkmelin

Comment 42

[Geoff Lankow (:darktrojan)]

Comment on attachment 9303238 [details]
Bug 1789437 - Remove ical.js "death by recursion" protection. r=#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: some recurring events (usually monthly, but others are possible) stop appearing beyond a certain point
Testing completed (on c-c, etc.): landed last week
Risk to taking this patch (and alternatives if risky): small, this is theoretically removing protection against an infinite loop but I'm not convinced that's even possible

Comment 43

[Wayne Mery (:wsmwk)]

Comment on attachment 9303238 [details]
Bug 1789437 - Remove ical.js "death by recursion" protection. r=#thunderbird-reviewers

[Triage Comment]
Approved for beta

Comment 44

[Daniel Darnell [:dandarnell]]

Thunderbird 108.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/526a06713d3f

Comment 45

[Geoff Lankow (:darktrojan)]

Comment on attachment 9303238 [details]
Bug 1789437 - Remove ical.js "death by recursion" protection. r=#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: some recurring events (usually monthly, but others are possible) stop appearing beyond a certain point
Testing completed (on c-c, etc.): in 108 beta 2
Risk to taking this patch (and alternatives if risky): small, this is theoretically removing protection against an infinite loop but I'm not convinced that's even possible

Comment 46

[Wayne Mery (:wsmwk)]

Comment on attachment 9303238 [details]
Bug 1789437 - Remove ical.js "death by recursion" protection. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr102

Comment 47

[Rob Lemley [:rjl]]

Thunderbird 102.6.0:
https://hg.mozilla.org/releases/comm-esr102/rev/caf6179c99bb