1790339 - caldav event corruption after Thunderbird upgrade from 91 to 102

Status: RESOLVED FIXED

(Calendar :: ICAL.js Integration, defect, P2)

Description

[Stéphane Chazelas]

Steps to reproduce:

Upgraded Thunderbird on Windows 10 from 91 to 102 (first corruption observed with 102.0.3, still happens if upgrading from 91 to 102.2.2) with Thunderbird profile configured with CalDAV calendars with offline support.

Actual results:

In the GUI, Some fields in events show as if there was a space in the middle of them like: "Attendee: te [e-mail]".

When users make modifications to those calendar events, such as removing a reminder, Thunderbird pushes a corrupted even to the calendar server (davical in our case). In most of the corruption cases we observed, the ics as stored on the server had things like:

ATTENDEE;CN=xxxx xxxxxxxx;ROLE=REQ-PARTICIPANT:mailto:xxxx.xxxxxxxx@xxxxxx\r\r\n \n x.com\r\n

(here showing the CR and LF characters as \r and \n respectively)

Though we've seen a few more types such as:

ATTENDEE;RSVP=TRUE;CN=xxxx xxxxxxxx;PARTSTAT=NEEDS-ACTION;ROLE=OPT-PARTICI\r\r\n ^n PANT:mailto:[e-mail]\r\n

It appears to have to do with ICS line folding.

I dug a bit and found that for those events, in the cache.sqlite file, here dumped with sqlite3 .dump, in 91 we could find things like:

INSERT INTO cal_attendees VALUES('529821cb-34af-4c90-849b-a7d4d70b8250',NULL,NULL,'9c66d062-ee28-4612-9569-73d31a7b0508',replace(replace('ATTENDEE;CN=Test User;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT:mailto:te\r\n [e-mail]\r\n','\r',char(13)),'\n',char(10)));

Where we do see the line folding inside the value as stored in the DB.

After upgrading to 102, that's still found there. However, if we start a clean cache in 102 (discarding the one from 91), we see:

INSERT INTO cal_attendees VALUES('529821cb-34af-4c90-849b-a7d4d70b8250',NULL,NULL,'a9c4ee29-b9cd-49b0-8ce7-b7c577409592',replace(replace('ATTENDEE;CN=Test User;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT:mailto:[e-mail]\r\n','\r',char(13)),'\n',char(10)));

For the same ICS, where line folding has been processed.

So my guess is that 102 is confused by those line foldings in the cache.sqlite inherited from 91.

The verified work around to prevent corruption is to untick "offline support" on all caldav servers before upgrading to 102.x (can be re-ticked afterwards).

That doesn't help with the corrupted events that were uploaded to the caldav server though.

That has all sorts of nasty side effects, users reporting missing events, not being able to discard reminders, not being able to modify events, etc.

Expected results:

No corruption.

Comment 1

[Geoff Lankow (:darktrojan)]

I can reproduce this, and I'm fairly certain about what caused it. Might not be easy to fix though.

Comment 2

[Geoff Lankow (:darktrojan)]

Okay, I have what I think is a reasonable solution. Still working out how to verify it works and doesn't cause other problems.

Comment 3

[Magnus Melin [:mkmelin]]

Dupe/xref bug 1774421.

Comment 4

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

Comment 6

[Sven Grull]

Any ideas in which release this regression fix will be included? What does severity S2 mean in terms of timeline?

Comment 7

[Geoff Lankow (:darktrojan)]

This will land in the next nightly build, I'll get it into beta ASAP, then after a week or two of running on beta we'll put it into the release. Probably 102.3.something or 102.4.0. In the meantime if it is a real pain you can switch the preference calendar.icaljs to false as a workaround.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/b27a0bf5a2ca
Unfold ICAL strings stored in calendar databases before use. r=mkmelin

Comment 9

[Geoff Lankow (:darktrojan)]

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: bad parsing of database entries leads to broken data
Testing completed (on c-c, etc.): landed 2 days ago
Risk to taking this patch (and alternatives if risky): small but non-zero

Comment 10

[Wayne Mery (:wsmwk)]

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Triage Comment]
Approved for beta

Comment 11

[Daniel Darnell [:dandarnell]]

Thunderbird 106.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/2ed2fc65d129

Comment 12

[Arthur K. (he/him)]

Regardless of the fix, for me, simply nuking the cache.sqlite while TB was closed and letting it re-create after launch made a big difference. The recreated cache.sqlite file went from 12.5MB to around 6.9MB. Odd. Could it have been that gummed up?

And a positive byproduct was that all the calendar events I had showing up on my work PC that were not showing up on my home PC are now there and vice versa. This was with 106.0b2.

Also of note, after TB recreated the cache.sqlite file, launching TB is nearly instant now (sub 1 second). It used to to take 2-3 seconds before I got the UI.

Comment 13

[Geoff Lankow (:darktrojan)]

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Approval Request Comment]
Regression caused by (bug #): ical.js
User impact if declined: bad parsing of database entries leads to broken data
Testing completed (on c-c, etc.): in 106.0b2
Risk to taking this patch (and alternatives if risky): small but non-zero

Comment 14

[Wayne Mery (:wsmwk)]

Comment on attachment 9294369 [details]
Bug 1790339 - Unfold ICAL strings stored in calendar databases before use. r=leftmostcat

[Triage Comment]
approved for esr102

Comment 15

[Rob Lemley [:rjl]]

Attachment: bug1790339_esr102.patch

[Triage Comment]
De-lazy-ified version for comm-esr102.

Comment 16

[Rob Lemley [:rjl]]

Thunderbird 102.3.2:
https://hg.mozilla.org/releases/comm-esr102/rev/6bf2a297b8a2