Closed Bug 1790339 Opened 3 months ago Closed 3 months ago

caldav event corruption after Thunderbird upgrade from 91 to 102

Categories

(Calendar :: ICAL.js Integration, defect, P2)

Thunderbird 102

Tracking

(thunderbird_esr102 fixed, thunderbird106 fixed)

RESOLVED FIXED
107 Branch
Tracking Status
thunderbird_esr102 --- fixed
thunderbird106 --- fixed

People

(Reporter: stephane+mza, Assigned: darktrojan)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files)

Steps to reproduce:

Upgraded Thunderbird on Windows 10 from 91 to 102 (first corruption observed with 102.0.3, still happens if upgrading from 91 to 102.2.2) with Thunderbird profile configured with CalDAV calendars with offline support.

Actual results:

In the GUI, Some fields in events show as if there was a space in the middle of them like: "Attendee: te st.user@example.com".

When users make modifications to those calendar events, such as removing a reminder, Thunderbird pushes a corrupted even to the calendar server (davical in our case). In most of the corruption cases we observed, the ics as stored on the server had things like:

ATTENDEE;CN=xxxx xxxxxxxx;ROLE=REQ-PARTICIPANT:mailto:xxxx.xxxxxxxx@xxxxxx\r\r\n \n x.com\r\n

(here showing the CR and LF characters as \r and \n respectively)

Though we've seen a few more types such as:

ATTENDEE;RSVP=TRUE;CN=xxxx xxxxxxxx;PARTSTAT=NEEDS-ACTION;ROLE=OPT-PARTICI\r\r\n ^n PANT:mailto:xxxx.xxxxxxxx@xxxxxxx.com\r\n

It appears to have to do with ICS line folding.

I dug a bit and found that for those events, in the cache.sqlite file, here dumped with sqlite3 .dump, in 91 we could find things like:

INSERT INTO cal_attendees VALUES('529821cb-34af-4c90-849b-a7d4d70b8250',NULL,NULL,'9c66d062-ee28-4612-9569-73d31a7b0508',replace(replace('ATTENDEE;CN=Test User;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT:mailto:te\r\n st.user@example.com\r\n','\r',char(13)),'\n',char(10)));

Where we do see the line folding inside the value as stored in the DB.

After upgrading to 102, that's still found there. However, if we start a clean cache in 102 (discarding the one from 91), we see:

INSERT INTO cal_attendees VALUES('529821cb-34af-4c90-849b-a7d4d70b8250',NULL,NULL,'a9c4ee29-b9cd-49b0-8ce7-b7c577409592',replace(replace('ATTENDEE;CN=Test User;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT:mailto:test.user@example.com\r\n','\r',char(13)),'\n',char(10)));

For the same ICS, where line folding has been processed.

So my guess is that 102 is confused by those line foldings in the cache.sqlite inherited from 91.

The verified work around to prevent corruption is to untick "offline support" on all caldav servers before upgrading to 102.x (can be re-ticked afterwards).

That doesn't help with the corrupted events that were uploaded to the caldav server though.

That has all sorts of nasty side effects, users reporting missing events, not being able to discard reminders, not being able to modify events, etc.

Expected results:

No corruption.

I can reproduce this, and I'm fairly certain about what caused it. Might not be easy to fix though.

Status: UNCONFIRMED → NEW
Ever confirmed: true
Regressed by: icaljs
Severity: -- → S2
Priority: -- → P2

Okay, I have what I think is a reasonable solution. Still working out how to verify it works and doesn't cause other problems.

Assignee: nobody → geoff
Status: NEW → ASSIGNED
Component: Provider: CalDAV → ICAL.js Integration

Dupe/xref bug 1774421.

See Also: → 1774421

Any ideas in which release this regression fix will be included? What does severity S2 mean in terms of timeline?

This will land in the next nightly build, I'll get it into beta ASAP, then after a week or two of running on beta we'll put it into the release. Probably 102.3.something or 102.4.0. In the meantime if it is a real pain you can switch the preference calendar.icaljs to false as a workaround.

Target Milestone: --- → 107 Branch

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/b27a0bf5a2ca
Unfold ICAL strings stored in calendar databases before use. r=mkmelin

Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: bad parsing of database entries leads to broken data
Testing completed (on c-c, etc.): landed 2 days ago
Risk to taking this patch (and alternatives if risky): small but non-zero

Attachment #9294369 - Flags: approval-comm-beta?

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Triage Comment]
Approved for beta

Attachment #9294369 - Flags: approval-comm-beta? → approval-comm-beta+

Regardless of the fix, for me, simply nuking the cache.sqlite while TB was closed and letting it re-create after launch made a big difference. The recreated cache.sqlite file went from 12.5MB to around 6.9MB. Odd. Could it have been that gummed up?

And a positive byproduct was that all the calendar events I had showing up on my work PC that were not showing up on my home PC are now there and vice versa. This was with 106.0b2.

Also of note, after TB recreated the cache.sqlite file, launching TB is nearly instant now (sub 1 second). It used to to take 2-3 seconds before I got the UI.

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: bad parsing of database entries leads to broken data
Testing completed (on c-c, etc.): in 106.0b2
Risk to taking this patch (and alternatives if risky): small but non-zero

Attachment #9294369 - Flags: approval-comm-esr102?

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Triage Comment]
approved for esr102

Attachment #9294369 - Flags: approval-comm-esr102? → approval-comm-esr102+

[Triage Comment]
De-lazy-ified version for comm-esr102.

Attachment #9297154 - Flags: approval-comm-esr102+
Attachment #9294369 - Flags: approval-comm-esr102+
Duplicate of this bug: 1789704
You need to log in before you can comment on or make changes to this bug.