1686466 - Task "Show" filtering broken in 78.6.1 - recurring tasks are all being marked incomplete

Status: VERIFIED FIXED

(Calendar :: Tasks, defect, P1)

Description

[Dave]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

No user changes

Actual results:

This bug was fixed last year but has returned in a different form in 78.6.1 (64-bit)
The task list "SHOW" filtering does not work correctly.
Also the Today pane task list in not filtering for the time period or task status.

Expected results:

Proper filter of date and task status

Comment 1

[Dave]

Looking at this a bit more.
Instances of recurring tasks are all being marked incomplete.
If I remark them as complete they will still revert to incomplete on restart of TB

I have a recurring task whose oldest instance is 1/1/2017
With 78.6.1 update now shown as incomplete - I mark it complete and it is sorted correctly - close TB - on reopen TB instance is listed as incomplete again.

Reinstall of TB did not fix.
Deactivate Calendar and reactive did not fix.
Restore of TB file data from before 78.6.1 (was 78.6.0) did not fix.

Comment 2

[drghughes]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1

I'm seeing the same thing. Started today, which I guess is after Thunderbird updated to 78.6.1, though as minor updates seem to occur silently these days, I can't be sure. It could also be due to the monthly Microsoft updates which occurred last night, but as @freggel.doe in bug 1686520 has the problem on Linux, this seems unlikely.

Comment 3

[drghughes]

Just to clarify, I'm seeing the problem with recurring tasks are all being marked incomplete, and if marked as complete, being shown as incomplete when Thunderbird is restarted.

Comment 4

[mzbugs]

Same behavior here: MAC OS X 10.12.6 Thunderbird 78.6.1 (64 bit)

Comment 5

[gbl954]

I'm seeing this as well in 85.b3 (64 bit). In addition, when I select a large number of these old recurring tasks and mark them complete, TB starts using excessive CPU and Memory (4GB) and becomes unresponsive. Win 10.19041.746. Restart did not fix it. Letting it run for an hour did not fix it.

Comment 6

[Richard Morris]

I'm seeing this as well. I was running Thunderbird 78.6.1 32-bit on Windows 10 when this started happening. This morning I tried to mark groups of 15 or so items completed and then Thunderbird would start freezing and showing as "not responding. Thunderbird then crashed. Now when ever I open Thunderbird, it just freezes/not responding and then crashes after about five minutes or so.

I uninstalled 78.6.1 and installed 78.6.0 but the behaviour is still the same. I can't use Thunderbird anymore as it just crashes after about five minutes every time I open it :(

I have managed to open 78.6.0 in safe-mode. It doesn't crash but it switches between moments of usefulness and moments of freezing/not responding.

Comment 7

[bugzilla]

Same behaviour here. TB 78.6.1 64-Bit on Windows 10.

I reverted my installation to TB 78.6.0 and the problem remained until I restored additionally the calendar files (calendar-data) from a backup.

Comment 11

[Magnus Melin [:mkmelin]]

So this is all with recurring tasks, correct? At least I don't see any direct problems with other tasks, but I don't use too much of recurring ones.

Comment 12

[mzbugs]

I would say, correct.

Comment 13

[Magnus Melin [:mkmelin]]

Can someone give more direct steps to reproduce? I created a recurring task in a local calendar now. I don't see anything wrong. Does it have to be caldav, or something else?
Please list what you do, and what the expectations are, as well how it differs from earlier.

Comment 14

[Magnus Melin [:mkmelin]]

If it regressed, it would likely be from bug 1664731.

Comment 15

[Dave]

(In reply to Magnus Melin [:mkmelin] from comment #11)

So this is all with recurring tasks, correct? At least I don't see any direct problems with other tasks, but I don't use too much of recurring ones.

Yes, recurring only, but overdue single tasks are still in green text, as I recall these were marked Red. Is that right?

Comment 16

[Dave]

(In reply to Magnus Melin [:mkmelin] from comment #13)

Can someone give more direct steps to reproduce? I created a recurring task in a local calendar now. I don't see anything wrong. Does it have to be caldav, or something else?
Please list what you do, and what the expectations are, as well how it differs from earlier.

The Start date must be in the past with tasks that were completed.

I updated some of my recurring tasks so that the Start date for the task is in the future and the past tasks are no longer shown as incomplete.

Comment 17

[Claus Berghammer]

(In reply to Magnus Melin [:mkmelin] from comment #13)

Can someone give more direct steps to reproduce? I created a recurring task in a local calendar now. I don't see anything wrong. Does it have to be caldav, or something else?
Please list what you do, and what the expectations are, as well how it differs from earlier.

Local Calendar. Create a recurring event for 1.1.2020, repeating weekly. This should give you 3 tasks. Mark the first two as completed than close TB and start again. The previously "completed" tasks are marked as incomplete again. But only with TB 78.6.1 (64 Bit build [Windows] here, don't know if that's relevant).

Comment 18

[mzbugs]

(In reply to Magnus Melin [:mkmelin] from comment #13)

Can someone give more direct steps to reproduce? I created a recurring task in a local calendar now. I don't see anything wrong. Does it have to be caldav, or something else?
Please list what you do, and what the expectations are, as well how it differs from earlier.

Just started Thunderbird after updating to 78.6.1 and there were hundreds of old (actually completed) task instance from all my recurring tasks shown in the tasks list of the "Tasks" tab and in the "Events & Tasks" pane of the tab with the e-mail accounts.

Even if I mark all this tasks anew as completed, they are not anymore shown as incomplete, but will be shown again as incomplete just by switching from e. g. "Incomplete Tasks" to "Current Tasks". They are than shown as incomplete in all list types selected.

Comment 19

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin [:mkmelin] from comment #14)

If it regressed, it would likely be from bug 1664731.

Comment 20

[Magnus Melin [:mkmelin]]

Following comment 17 I can reproduce, and can also confirm backing out https://hg.mozilla.org/comm-central/rev/95c92a2f9fbc (bug 1664731) fixes it.

For me, the first occurrence is correctly marked as completed, it's just the second (and onwards I guess) that refuses to comply.

Likely not the cause, but when shutting down I get (before that patch too it seems):

console.error: Lightning: 
  [calStorageCalendar] Message: prepareStatement exception
Last DB Statement: [object AsyncStatementJSHelper]
Exception: [Exception... "Component not initialized"  nsresult: "0xc1f30001 (NS_ERROR_NOT_INITIALIZED)"  location: "JS frame :: resource:///modules/CalStorageCalendar.jsm :: prepareStatement :: line 236"  data: no]
1: [resource:///modules/CalStorageCalendar.jsm:2589] logError
2: [resource:///modules/CalStorageCalendar.jsm:239] prepareStatement
3: [resource:///modules/CalStorageCalendar.jsm:1744] _assureRecurringItemCaches
4: [resource://gre/modules/AsyncShutdown.jsm:573] observe

JavaScript error: resource:///modules/CalStorageCalendar.jsm, line 288: TypeError: can't access property "executeAsync", this.mDB is null

Comment 21

[Anje]

Confirm there is a problem. I can reproduce.

For me, the first occurrence is correctly marked as completed, it's just the second (and onwards I guess) that refuses to comply.

Confirm I see the same.

I'm running some various tests to help nail down issue.

TEST:
In Task tab.
Select earliest task, click delete, select delete all occurrances. Removes all.
REsult :
After restart remain deleted. deletion does work when using original to delete all.

In TASK tab
Select one of the reoccuring tasks - not the original and delete.
Result: after restart remains deleted. So this works.

Tested:
Oldest/original is set as completed - this works as expected.
In TASK TAB
Next oldest - set as 75 % complete
also tested manually - Edit only this occurance
REsult: After restart this setting is also removed in both instances - so a fail.

Tested:
Select original > edit all occurances - manually set Status as completed on today date 100% complete. SAve and close.
REstart Thunderbird - all occurances still set as completed.
So manually editing the original Task in it's own window for all occurances does work as expected.

Comment 22

[msedv - Markus Schwaiger]

Remark: same behaviour when using caldav (ownCloud) instead of local calendar.

Terrible if haveing used recurring tasks for years - my whole "personal workflow" is killed.

Comment 23

[mzbugs]

Perhaps a silly question.
78.6.1 seems to be a minor Update/Bugfix. Why not reverse the calendar handling code from 78.6.1 to 78.6.0?
The calendar worked almost perfectly in 78.6.0. (there where always some minor bugs, e. g. you had to click the category selector and entry two times to select a category for a task).

Meanwhile I got back to version 78.6.0 and fortunately I had a backup of the calendar three weeks prior to the 78.6.1 Update. I didn't create a lot of new tasks in between. So I could use the old calendar data in 78.6.0.

It didn't work to use my current calendar data from 78.6.1 with 78.6.0. Same bug with "completed/not completed" there. I explicitly had to use my last saved calendar data form 78.6.0.

Comment 24

[Magnus Melin [:mkmelin]]

Because this bug was caused by fixing bug 1664731 which was a likely(?) even larger nuisance.

Lasana, any update on figuring this one out?

Comment 25

[gbl954]

Attachment: Error console

Another data point:
Coming out of suspension, TB spikes CPU usage and Memory and is unresponsive for several minutes

Comment 26

[Lasana Murray]

(In reply to Magnus Melin [:mkmelin] from comment #24)

Because this bug was caused by fixing bug 1664731 which was a likely(?) even larger nuisance.

Lasana, any update on figuring this one out?

No not yet.

Comment 27

[Lasana Murray]

(In reply to Magnus Melin [:mkmelin] from comment #20)

Following comment 17 I can reproduce, and can also confirm backing out https://hg.mozilla.org/comm-central/rev/95c92a2f9fbc (bug 1664731) fixes it.

For me, the first occurrence is correctly marked as completed, it's just the second (and onwards I guess) that refuses to comply.

Likely not the cause, but when shutting down I get (before that patch too it seems):

console.error: Lightning: 
  [calStorageCalendar] Message: prepareStatement exception
Last DB Statement: [object AsyncStatementJSHelper]
Exception: [Exception... "Component not initialized"  nsresult: "0xc1f30001 (NS_ERROR_NOT_INITIALIZED)"  location: "JS frame :: resource:///modules/CalStorageCalendar.jsm :: prepareStatement :: line 236"  data: no]
1: [resource:///modules/CalStorageCalendar.jsm:2589] logError
2: [resource:///modules/CalStorageCalendar.jsm:239] prepareStatement
3: [resource:///modules/CalStorageCalendar.jsm:1744] _assureRecurringItemCaches
4: [resource://gre/modules/AsyncShutdown.jsm:573] observe

JavaScript error: resource:///modules/CalStorageCalendar.jsm, line 288: TypeError: can't access property "executeAsync", this.mDB is null

This looks like bug 1671051. As I mentioned, there is likely to be more bugs like it lurking around.

Comment 28

[Werner Hoelzl]

Is this bug fixed in Thunderbird 78.7?

Comment 29

[Magnus Melin [:mkmelin]]

No, this bug is still NEW. Once we find a fix it will get RESOLVED. Then backported to beta and, once the thunderbird_esr78 flag is set to fixed, it's in the next point release.

Comment 30

[Werner Hoelzl]

thanks

Comment 31

[Lasana Murray]

This looks like a caching issue. I have a patch coming soon.

Comment 32

[Lasana Murray]

Attachment: bug1686466.patch

From what I can tell, the modified recurring items have their parent item displayed instead of the exception. Looking at CalStorageCalendar again it seems that the get(Event|Todo)FromRow() method caches items automatically. The cache is checked by id which is the same for all occurrences of an event.

For now, I added the option to skip caching and updated the test to detect this.

Comment 33

[drghughes]

I've installed Thunderbird Beta along side the release version. Is there anything that I can do to test this patch, or do I need to wait for it to make its way into a beta?

Comment 34

[Lasana Murray]

(In reply to [e-mail] from comment #33)

I've installed Thunderbird Beta along side the release version. Is there anything that I can do to test this patch, or do I need to wait for it to make its way into a beta?

The patch needs to be reviewed then approved before landing. If you are comfortable, you can build from source and apply this patch directly.
It may be better to wait until this lands though.

Comment 35

[Geoff Lankow (:darktrojan)]

Comment on attachment 9199851 [details] [diff] [review]
bug1686466.patch

Review of attachment 9199851 [details] [diff] [review]:
-----------------------------------------------------------------

The test changes are good but the change to CalStorageCalendar itself should be an early return in cacheItem if item.recurrenceId is truthy. Items that are recurrences or exceptions should never be cached and returning early will prevent that in all current and future uses of cacheItem.

Comment 36

[codename]

It looks like you know that Bug 1664731 is [now] no longer fixed.

Comment 37

[Lasana Murray]

Attachment: bug1686466v2.patch

Changed cacheItem to skip caching items with a recurrenceId.

Comment 38

[Lasana Murray]

Try run:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=0dfd995fb1b2514733eb48bfc05d4a1a03468fc6

Comment 39

[Geoff Lankow (:darktrojan)]

Comment on attachment 9200656 [details] [diff] [review]
bug1686466v2.patch

Review of attachment 9200656 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/31cf458c7526
Do not cache items with a recurrenceId in CalStorageCalendar. r=darktrojan

Comment 41

[Magnus Melin [:mkmelin]]

Comment on attachment 9200656 [details] [diff] [review]
bug1686466v2.patch

[Approval Request Comment]
Regression caused by (bug #): 1664731
User impact if declined: problems with recurring tasks (and events)
Testing completed (on c-c, etc.): on c-c
Risk to taking this patch (and alternatives if risky): Seems hard to estimate, but it's tested. Should go through beta.

Comment 42

[Wayne Mery (:wsmwk)]

Comment on attachment 9200656 [details] [diff] [review]
bug1686466v2.patch

[Triage Comment]
Approved for beta

Comment 43

[Glen A.]

When will this be available in a patch release (78.7.1)?

Comment 44

[Magnus Melin [:mkmelin]]

It will first go to beta, so assuming things go well 78.8 in a little more than 2 weeks.

Comment 46

[Dave]

Not sure if it is related but I noticed it at the same time - Overdue tasks in the Today Pane are in green text, as I recall they were marked with red text.

Comment 47

[Rob Lemley [:rjl]]

Thunderbird 86.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/29fb09a37f4f

Comment 48

[drghughes]

In Thunderbird 86.0b2 I've been able to mark recurring tasks as complete and have them remain complete after a restart. This was with tasks that were created with v78.6 and imported into a new profile that I am using for the beta.

I have also created a new recurring task that I will check over the next few days.

Comment 49

[codename]

Just FYI - I have noticed that (currently with 78.7.1) when I create a new recurring/repeating event (say monthly on 3rd Tuesday) I can edit the first event and it will remain correctly edited, but any other future events which are edited revert back to the original [unedited] event description.

Comment 50

[freggel.doe]

(In reply to codename from comment #49)

Just FYI - I have noticed that (currently with 78.7.1) when I create a new recurring/repeating event (say monthly on 3rd Tuesday) I can edit the first event and it will remain correctly edited, but any other future events which are edited revert back to the original [unedited] event description.

As was mentioned above in comment 44: if all goes well this might get fixed in 78.8.

Comment 51

[codename]

(In reply to freggel.doe from comment #50)

(In reply to codename from comment #49)

Just FYI - I have noticed that (currently with 78.7.1) when I create a new recurring/repeating event (say monthly on 3rd Tuesday) I can edit the first event and it will remain correctly edited, but any other future events which are edited revert back to the original [unedited] event description.

As was mentioned above in comment 44: if all goes well this might get fixed in 78.8.

I have been involved with this bug by reporting it as a problem on bugzilla many months ago. My comment #49 was intended to inform those attempting to fix this problem not to test (by editing) just the first or original event date in which the recurring event starts, but to edit recurring dates further out (date-wise) from the starting date to make sure that editing of only that occurrence remains edited and does not revert back to the original description.

Comment 52

[gbl954]

Attachment: TB after hibernation.png

Testing 86b2 as well. Completed recurring task instances stay completed, however, marking a single instance completed still takes an enormous amount of resources and a long time (up to a minute). Also coming out of hibernation, TB becomes unresponsive and takes enormous amount of resources for several minutes (see attached)
.

Comment 53

[gbl954]

Attachment: Reminder TB86b2.png

Another anomaly is with reminders - single event creates multitude of reminders (see attached).

Part of the issue appears to be having lots of occurrences - e.g. a couple of weekly tasks dating back to 2013 (starting dates) cause tremendous slowdown. I've reduced resource requirements by resetting starting dates on those tasks to a more recent date range (Jan2021).

Comment 54

[Lasana Murray]

(In reply to gbl954 from comment #53)

Created attachment 9202366 [details]
Reminder TB86b2.png

Another anomaly is with reminders - single event creates multitude of reminders (see attached).

Part of the issue appears to be having lots of occurrences - e.g. a couple of weekly tasks dating back to 2013 (starting dates) cause tremendous slowdown. I've reduced resource requirements by resetting starting dates on those tasks to a more recent date range (Jan2021).

Please file another bug for the performance issues you are having. Feel free to cc me. While most likely related, comments on new issues here make it difficult to follow the analysis and resolution. Thanks!

Comment 55

[drghughes]

Attachment: Demonstration of change to single recurring event - see comment 51

(In reply to codename from comment #51)

I have been involved with this bug by reporting it as a problem on bugzilla many months ago. My comment #49 was intended to inform those attempting to fix this problem not to test (by editing) just the first or original event date in which the recurring event starts, but to edit recurring dates further out (date-wise) from the starting date to make sure that editing of only that occurrence remains edited and does not revert back to the original description.

I've tested changing a single task in a set of recurring tasks in Thunderbird 86.0b2, and restarted, and the change was still there on restart as the screenshot shows.

Comment 56

[drghughes]

Everything is looking good with this fix. Is there anything that you want me to test?

Comment 57

[drghughes]

Attachment: Changing all occurrences of a recurring task - completed task not changed

Just tried changing all occurrences of a recurring task, including one occurrence that had already been marked as completed in Thunderbird 86.0b2.

The uncompleted tasks were changed (Flutracking to Flu tracking), but the completed task was not changed.

That approach probably makes sense, but reporting it for completeness.

Comment 58

[codename]

Maybe irrelevant but, should editing recurring 'Events' be tested separately instead of just 'Tasks'?

What I find odd is that with the v78.6.1 update (on January 12th) I believe this problem was fixed. I quickly learned that the fix only applied to newly originated (current date or later) events because old existing events (originated prior to v78.6.1 or the 1-12-2021 fix) in which a future event was edited did not retain the editing and reverted back to the original description. But for some reason that 1-12-2021 fix (for newly originated recurring events in which future event dates were edited) suddenly stopped working - maybe it stopped working with the 78.7.0 update...dunno.

Comment 60

[JNH]

When can we expect 78.8.0 and a fix of this anoing problem?

Comment 61

[Magnus Melin [:mkmelin]]

78.8 is scheduled for Feb 23.

Comment 62

[Wayne Mery (:wsmwk)]

Comment on attachment 9200656 [details] [diff] [review]
bug1686466v2.patch

[Triage Comment]
Approved for esr78

Comment 64

[Rob Lemley [:rjl]]

Thunderbird 78.8.0:
https://hg.mozilla.org/releases/comm-esr78/rev/2c00a9789fb2

Comment 66

[Dave]

Working here on 78.8.0 (64-bit).
Thank you all!

Comment 67

[Anje]

Downloaded and tested - Windows OS - Thundrebird 78.8.0 (64-bit).
All seems to be working correctly.
I'm getting reports in Support forum that all seems to OK.
On behalf of many people in Support Forum - thanks to all concerned for the rapid response and fix.

Comment 68

[mzbugs]

Just updated to 78.8.0 (64-bit) on Mac OS 10.14.6. Everything is fine again so far. Thanx :). Love Thunderbird :).

Comment 69

[George]

I upgraded to 78.8.0 (64-bit) on Windows 10
-It actually marks as complete the recurring tasks.

• However , TB hangs up when trying to load the calendar (in fact during startup). I have a large list of tasks and events. \local.sqlite is ~50MB. If I removed the \local.sqlite , everything works fine and TB starts normally. Any work around?

Comment 70

[Joffe Gruenfron]

Tested TB 78.8.0 (64-bit) on Linux and can confirm that bug 1692181 is fixed.
Thanks!

Comment 71

[gbl954]

(In reply to George from comment #69)

I upgraded to 78.8.0 (64-bit) on Windows 10
-It actually marks as complete the recurring tasks.

• However , TB hangs up when trying to load the calendar (in fact during startup). I have a large list of tasks and events. \local.sqlite is ~50MB. If I removed the \local.sqlite , everything works fine and TB starts normally. Any work around?

same performance issues here, my local.sqlite is also ~54MB. I was asked to start a new bug 1692001.

Comment 72

[George]

Thanks. I will move to the other thread.