1664731 - After release 78 some event descriptions and categories are lost when editing a single recurring event

Status: VERIFIED FIXED

(Calendar :: General, defect)

Description

[RaymondR]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0

Steps to reproduce:

I have many repeating calendar event. If I "Edit all occurrence" for one of these repeating events I see the description that was originally entered when the repeating event was created.
If I "edit only this occurrence" then the description is now always blank.

Actual results:

The description is blank.

Expected results:

This should work the same as it did in release 68. That is, when "edit only this occurrence" is selected you should see the description for that occurrence. When "edit only this occurrence" is used for the first time the description for the "all occurrences should be carried into the "only this occurrence"
For me, this is a huge problem.

Comment 1

[Wayne Mery (:wsmwk)]

I'm not aware of single version 78 report - all of https://mzl.la/2RulvKl well predate 78

Comment 2

[RaymondR]

Further to my original report.

This happens when the original repeating event was created on rel 68 and then one of its occurrences was "edit only this occurrence" on rel 68. When rel 78 is installed, the description in the edited occurrence is blank.

To make matters worse, you apparently can't de-install rel 78 and go back to rel 68 because rel 78 changes something in the profile which is not backwards compatible to rel 68.

Comment 3

[Wayne Mery (:wsmwk)]

regression?

Comment 4

[Paul Morris [:pmorris]]

Sounds like a regression. Not sure what would be causing this. Perhaps something going wrong in the upgrade, based on comment 2. I checked on current trunk and could not reproduce the issue there (I don't have a 68 build locally to try and reproduce).

RaymondR, what kind of calendar(s)? Local, CalDAV, Google? If Google are you using the Google provider add-on? If they are network calendars is there a way to access them on the web to see if the descriptions have been lost or are still there and are just not being displayed properly?

Comment 5

[RaymondR]

In this case, the repeating events are all in local calendars.

Comment 6

[RaymondR]

Another piece of information.

If I do "edit only this occurrence" under rel 78, the updates appear to be there as expected. HOWEVER, if I then shutdown Thunderbird and restart it, all of the "edit only this occurrence" changes disappear.

Comment 7

[RaymondR]

One more piece of information although I don't think this is related

I have moved the profile directory out of is default location and put it into Dropbox. I don't remember exactly how I did this but its been working fine for 4 or 5 years. I do notice now that there is a partial profile that exists back in the default location under User->Local->AppData->Thunderbird

Comment 8

[RaymondR]

How I moved the directory.

I moved the Profile from User->AppData->Roaming->Thunderbird->Profiles into my Dropbox location. I then built a Windows "Junction" in the old location that points to the real location in Dropbox. As previously noted, this has worked fine for 4 or 5 years but I do acknowledge that Windows Junctions can be problematic.

Comment 9

[RaymondR]

Further to my previous notes. If I "edit only this occurrence" and change the description, all initially looks well until I shutdown and restart Thunderbird. When I restart it, not only is the description gone but also the "reminder" is cleared to "no reminder" and will never stay set on again.

I desperately need to get a bypass or a fix for this. Can you make any suggestions?

Comment 10

[RaymondR]

P.S. If there is any additional diagnostic info I can provide please let me know. I am an experienced developer albeit a little dated at this instant.

Comment 11

[codename]

Raymond - I have the exact same problem. I reported it myself here: https://bugzilla.mozilla.org/show_bug.cgi?id=1674876

You may want to read how I reported the problem.

Maybe it will be correctly now that there are multiple reports of the same problem.

Comment 12

[RaymondR]

Codename - Sounds like we have duplicates. This is great news. Hopefully the team will be able to make some progress now. I have been looking for environmental issues that might cause it but other than what I've already noted above I haven't been able to determine any such cause.

Comment 13

[codename]

Yep! Let's hope the Tb Calendar guru experts get this problem taken care of quickly.

I also wish they would provide a means to see which events are repeating events, and also which repeating events have been modified since they were originally created. The 'Calendar Tweaks' add-on provided an icon on the event bar for repeating events and a slash (/) through the icon if the repeating events had been modified. This feature needs to be added to the Calendar.

Here is my 'Enhancement" requests for the repeating icon: https://bugzilla.mozilla.org/show_bug.cgi?id=1674912

Comment 15

[codename]

Magnus - I see that you have cancelled my bug report (1674876) for this exact same problem. I presume this bug (by RaymondR) will be the bug that will remain open until it is resolved, correct?

Comment 16

[Magnus Melin [:mkmelin]]

Yes, we only need one bug open to track the same issue.

Comment 17

[Anje]

Raymond
re:To make matters worse, you apparently can't de-install rel 78 and go back to rel 68 because rel 78 changes something in the profile which is not backwards compatible to rel 68.
There is a specific method to do a downgrade on same profile.
https://support.mozilla.org/en-US/questions/1311576#answer-1363839

Comment 19

[WaltS48 [:walts48]]

Thanks to the more detailed STR provided by Code Name in a mozillaZine post. I can reproduce this following these STR using Thunderbird 78.5.0 on Ubuntu 18.04 and 20.04 LTS Linux.

Working properly in my Ubuntu supplied 68.10.0.

• Assuming you already use the Calendar - just create a repeating event (say either weekly or monthly), give it a Title and select a Category. Type a >short message in the description box.
• Save it. Exit Tb.
• Restart Tb and open the Calendar
• Go to any one of the repeating events that you just created and use "Edit only this occurrence".
• Type an added line in the description box (which is your modification for 'only' that one date of the repeating event). Save it.
• Exit Calendar and exit Tb.
• Restart Tb and open Calendar.
• Go to that event date that you just modified and you will see the entire description box is blank - no longer having the original message nor the >added [modified] message (if it is reacting like what we have seen). Also, the event's selected Category is lost and becomes 'None'.

Comment 20

[Magnus Melin [:mkmelin]]

Lasana, please see if you can track this one down.

Comment 24

[Lasana Murray]

I can see the event and its properties in the sqlite database but it looks like they are not loaded for some reason after restart.

Comment 25

[Lasana Murray]

Attachment: bug1664731.patch

This bug happens because of the way the sqlite backed calendar loads items.

The CalStorageCalendar.getItems() method loads properties for regular and recurring events but not for edited recurring events (exceptions). That's because false is passed to the second parameter of getEventFromRow/getEventFromTodo. The reason this bug is only noticed on restart is because modifications to items are cached so the issue only becomes apparent when a new instance is created.

Comment 26

[Lasana Murray]

CalStorageCalendar seems to be doing way too much, it's hard to figure out what's going on. Maybe the actual SQL operations could be moved out into a separate class as a start?

Comment 27

[Lasana Murray]

Try run in progress:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=fefa5907086e1e26b98da0dc11ba46bb4925ba95

Comment 28

[Geoff Lankow (:darktrojan)]

Comment on attachment 9192766 [details] [diff] [review]
bug1664731.patch

Review of attachment 9192766 [details] [diff] [review]:
-----------------------------------------------------------------

I'm happy with this in general but I'd like you to rearrange the test.

::: calendar/test/unit/test_storageGetItems.js
@@ +105,5 @@
> +  ];
> +
> +  // Whenever we get the storage calendar we need to request a profile,
> +  // otherwise the cleanup functions will not run
> +  do_get_profile();

This can go at the top with the imports.

@@ +108,5 @@
> +  // otherwise the cleanup functions will not run
> +  do_get_profile();
> +
> +  let work = [eventData, todoData];
> +  for (let [filterType, originalItem, originalProps, changedProps] of work) {

Pull this loop into a function of its own:

function subtest(filterType, originalItem, originalProps, changedProps) { … }

Then just call it for each item type, rather than creating an array of arguments to pull apart again later. If you do each item type in a new task function, that makes debugging a bit easier too.

@@ +109,5 @@
> +  do_get_profile();
> +
> +  let work = [eventData, todoData];
> +  for (let [filterType, originalItem, originalProps, changedProps] of work) {
> +    let isTodo = filterType & Ci.calICalendar.ITEM_FILTER_TYPE_TODO;

You only use `isTodo` in one place, this line can go down there.

::: calendar/test/unit/xpcshell-shared.ini
@@ +46,5 @@
>  [test_rfc3339_parser.js]
>  [test_search_service.js]
>  [test_startup_service.js]
>  [test_storage.js]
> +[test_storageGetItems.js]

Let's have it match all the others: test_storage_getitems.js

Comment 29

[Lasana Murray]

Attachment: bug1664731v2.patch

Changes made.

Comment 30

[Lasana Murray]

Latest try run:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&selectedTaskRun=Ndl7qdItTBidpQLAAH0apA.0

Comment 31

[Geoff Lankow (:darktrojan)]

Comment on attachment 9193286 [details] [diff] [review]
bug1664731v2.patch

That looks better. Thanks.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/95c92a2f9fbc
Load properties for exceptions of recurring events and todos. r=darktrojan

Comment 36

[Magnus Melin [:mkmelin]]

Comment on attachment 9193286 [details] [diff] [review]
bug1664731v2.patch

[Approval Request Comment]
Regression caused by (bug #): bug 501689
User impact if declined: loss of event data.

Lots of duplicates for this.

Comment 37

[Wayne Mery (:wsmwk)]

Comment on attachment 9193286 [details] [diff] [review]
bug1664731v2.patch

[Triage Comment]
Approved for beta

Comment 38

[Rob Lemley [:rjl]]

Thunderbird 85.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/2d31cd452e1e

Comment 40

[Wayne Mery (:wsmwk)]

Comment on attachment 9193286 [details] [diff] [review]
bug1664731v2.patch

[Triage Comment]
Approved for esr78

Comment 41

[Rob Lemley [:rjl]]

Thunderbird 78.6.1:
https://hg.mozilla.org/releases/comm-esr78/rev/296067b63ef5

Comment 43

[Geoff Lankow (:darktrojan)]

Attachment: 1664731-backport-esr78.diff

This fixes the broken test on 78.

Comment 44

[Wayne Mery (:wsmwk)]

Comment on attachment 9195767 [details] [diff] [review]
1664731-backport-esr78.diff

[Triage Comment]
Approved for esr78

Comment 45

[Rob Lemley [:rjl]]

Thunderbird 78.6.1:
https://hg.mozilla.org/releases/comm-esr78/rev/022d4d7bf7b2

Comment 46

[WaltS48 [:walts48]]

Verifying in the 78.6.1 rc on Windows 10.

Comment 48

[codename]

Thank you!
It appears the fix only applies to newly added repeating events (created after this fix) in which a single event is edited ("Edit only this occurrence"). It appears the fix does not take care of existing (before the fix) repeating events in which a single event [either] had been edited or is now edited. In other words, if you had a repeating event that was created prior to this fix - editing a single repeating event does not work. Only newly created repeating events can have editing of a single occurrence...

Comment 49

[Magnus Melin [:mkmelin]]

Yes old data would have been lost.

Comment 50

[codename]

I have noticed that the correct (original) category is brought back from the 'None' category when it was broken.

Also, prior to this problem arising once you edited a repeating occurrence, if you wanted to edit that same occurrence again you were not asked if you wanted to 'Edit only this occurrence' or 'Edit all occurrences'. Instead, when you clicked on an edited event it automatically opened without asking whether you wanted to 'Edit only this occurrence' or 'Edit all occurrences' because that particular event had already been edited.

I presume the new way (offering a choice to edit only this occurrence or edit all occurrences) is the way you want it, but it seems to me that once you have edited an occurrence and you want to edit that same occurrence again it's best to not have the choice because it is not needed and it can be confusing to have the choice. This, in my opinion, is why it would be very helpful to add an enhancement to indicate (with an icon on the event bar) that an event is a repeating event...and a slightly different icon with a slash (/) through it to indicate that a repeating event has been edited since it was created. This was reported as an enhancement suggestion (Bug 1674912) here: https://bugzilla.mozilla.org/show_bug.cgi?id=1674912

Comment 52

[Joffe Gruenfron]

Question: since this fix (1664731v2.patch) I encounter problems with calendar recurring events and filed Bug 1692181 for this. I can confirm that only this patch is responsible for the problem. Can anyone point out what and why this happens?

Comment 54

[Craig Stevens]

I accepted a set of repeating meeting events in TB 78 for my Google calendar. Every time the calendar syncs, which is every 30 mins, I get alerts for yesterday's meeting and anything for today that's in the past. I can't dismiss them.

It seems to me I'm getting the Google calendar alert AND an alert from the Home calendar, which doesn't hold the events anyway. The reason I say that is because the first dialogue is what I'd expect, just a pop up listing two items and asking if I want to dismiss individually or all. Then I get a red taskbar panel icon appear and a pop up of a different format which again won't let me dismiss it. They all have to be closed as windows. I'm using Linux MInt 20.1 Cinnamon. I have TB 78.7.1 (64-bit) with the "Provider for CalDav & CardDAV" and "TBSync" extensions installed.

It's only been happening since I accepted the repeating events 9 days ago. It's a complete PITA.

Comment 55

[Arthur K. (he/him)]

(In reply to Craig Stevens from comment #54)

I accepted a set of repeating meeting events in TB 78 for my Google calendar. Every time the calendar syncs, which is every 30 mins, I get alerts for yesterday's meeting and anything for today that's in the past. I can't dismiss them.

It seems to me I'm getting the Google calendar alert AND an alert from the Home calendar, which doesn't hold the events anyway. The reason I say that is because the first dialogue is what I'd expect, just a pop up listing two items and asking if I want to dismiss individually or all. Then I get a red taskbar panel icon appear and a pop up of a different format which again won't let me dismiss it. They all have to be closed as windows. I'm using Linux MInt 20.1 Cinnamon. I have TB 78.7.1 (64-bit) with the "Provider for CalDav & CardDAV" and "TBSync" extensions installed.

It's only been happening since I accepted the repeating events 9 days ago. It's a complete PITA.

This seems to be the fault of extensions. I used to have this exact issue when I used Provider for Google Cal extension but after switching to using the built-in CalDAV functionality of Lightning, all my problems went away. I reported a similar issue in bug 1601294.

Comment 56

[Craig Stevens]

I was originally using the "Provider for Google Calendars" but I switched to "Provider for CalDav & CardDAV" as the former was failing to subscribe to calendars all of a sudden as, while testing, I unsubscribed and resubscribed to my Google calendar and found I couldn't. It also seems Google users have upset the TBSync maintainer and so they put support as experimental now, which isn't good. Wrt Lightning, my understanding is that it isn't used any longer. The package is incompatible with TB 78 hence having to use Provider for "CalDav & CardDAV". I'm finding it all a bit of a mess and really wish TB would add calendar subscription functionality for the 3 or 4 major calendar providers out of the box.

Comment 57

[Arthur K. (he/him)]

(In reply to Craig Stevens from comment #56)

I was originally using the "Provider for Google Calendars" but I switched to "Provider for CalDav & CardDAV" as the former was failing to subscribe to calendars all of a sudden as, while testing, I unsubscribed and resubscribed to my Google calendar and found I couldn't. It also seems Google users have upset the TBSync maintainer and so they put support as experimental now, which isn't good. Wrt Lightning, my understanding is that it isn't used any longer. The package is incompatible with TB 78 hence having to use Provider for "CalDav & CardDAV". I'm finding it all a bit of a mess and really wish TB would add calendar subscription functionality for the 3 or 4 major calendar providers out of the box.

Not to belabor the discussion but Lightning was integrated into TB a while ago so not a separate extension as you allude to. It's all under the hood now. I myself am using TB 88 beta and much of the CalDAV integration / support is >far< superior to what's in 78 and earlier. But I am not suggesting to try the beta for any quick & dirty fix. Much of setup pain and functionality of using the calendar is going to make, IMHO, many calendar users happy.

I also know much of the CardDAV support is being ironed out and should see the light of day for the TB 91 release (which is not too far out. Maybe July? See https://www.ghacks.net/2021/02/10/thunderbird-team-outlines-plans-for-the-email-client/) for all the details.

Comment 58

[Craig Stevens]

(In reply to Arthur K. [He/Him/His] from comment #57)

Not to belabor the discussion but Lightning was integrated into TB a while ago so not a separate extension as you allude to. It's all under the hood now. I myself am using TB 88 beta and much of the CalDAV integration / support is >far< superior to what's in 78 and earlier. But I am not suggesting to try the beta for any quick & dirty fix. Much of setup pain and functionality of using the calendar is going to make, IMHO, many calendar users happy.

I also know much of the CardDAV support is being ironed out and should see the light of day for the TB 91 release (which is not too far out. Maybe July? See https://www.ghacks.net/2021/02/10/thunderbird-team-outlines-plans-for-the-email-client/) for all the details.

That's very good to know. Thanks :)