1843405 - Calendar sync is broken on Fastmail, showing only disabled calendars with 'enable' next to them

Status: NEW

(Calendar :: Provider: CalDAV, defect, P2)

Description

[jvisser]

Attachment: Screenshot 2023-07-13 204326.png

Steps to reproduce:

• Cleanly install Thunderbird 115
• Fill in my e-mail account name and (app) passsword
• Fill in the auth window that Fastmail pops up

Actual results:

• E-mail sync, but all calendar sync is broken

Expected results:

• Calendars should sync

Comment 1

[Wayne Mery (:wsmwk)]

Ed, does this reproduce for you?

Comment 2

[Ed]

I don't use calendars, but I did a test:

• made an entry on TB v116

• made an entry on TB v102.13

• made an entry on Fastmail via the web interface

Result: none of the entries synced anywhere.

Comment 3

[luniac]

I can confirm the bug. I'm fairly sure it's linked to the new oauth support for fastmail which was released last month. Unfortunately there doesn't seem to be a way to use the app-specific passwords anymore. Attempting to use them displays a message that basically says you need to use your main login credentials (i.e. for oauth).

I turned on verbose logging and I was getting a bunch of errors, e.g.

Could not perform playback operation modify for item
CalDAV: Error: got status 401 fetching calendar data

It kept on trying to replay a dismiss notification event to fastmail's caldav server but for whatever reason it failed and triggers thunderbird to disable the calendar. Manually editing the calendar on fastmail's site (to remove the event reminders) by itself didn't help. I had to remove all of my fastmail calendars from thunderbird and re-add them.

This seemed to fix the problem, although while typing this message, one of my calendars died again (disabled itself). This was in the logs:

[calCachedCalendar] replay action failed: <unknown>, uri=https://caldav.fastmail.com/dav/calendars/user/<removed>, result=2147500037, operation=[xpconnect wrapped calIOperation]

Not sure what it is trying to replay. The calendar should be fresh as far as thunderbird is concerned as I readded the calendar...

Either way, this is clearly a regression. Can we get a comment from the dev team?

Comment 4

[Ed]

The basic Fastmail setup instructions say nothing about OAUTH; I'm still using an app PW.

https://www.fastmail.help/hc/en-us/articles/1500000278342

Looks like OAUTH is only necessary for sending from other addresses.

https://www.fastmail.help/hc/en-us/articles/4409885100431-Sending-from-other-addresses-OAuth2

Comment 5

[luniac]

(In reply to Ed from comment #4)

The basic Fastmail setup instructions say nothing about OAUTH; I'm still using an app PW.

https://www.fastmail.help/hc/en-us/articles/1500000278342

Looks like OAUTH is only necessary for sending from other addresses.

https://www.fastmail.help/hc/en-us/articles/4409885100431-Sending-from-other-addresses-OAuth2

You must be using an old version of Thunderbird then. Whenever I try to add a fastmail calendar, it forces me to use oauth login flow and won't let me use an app-specific password.

Comment 6

[luniac]

Setup:

• Thunderbird 115.1.0
• 3 Fastmail calendars configured
• Have an event with a reminder active (i.e. waiting to be dismissed or snoozed)

I did some more testing and I was able to force thunderbird to use an app specific password. It required me to close the oauth authorisation flow twice in row.

However this did not end up helping as I'm still having errors with one of my calendars when I try to dismiss a notification, resulting in the calendar going into a read-only state.

[calICSCalendar] Backup failed, no copy: TypeError: oldFile is undefined

There has been an error reading data for calendar: xxxx. It has been placed in read-only mode, since changes to this calendar will likely result in data-loss. You may change this setting by choosing 'Edit Calendar'. Error code: DAV_PUT_ERROR. Description: Publishing the calendar file failed
Status code: 0

An error occurred when writing to the calendar xxxx! Please see below for more information. Error code: MODIFICATION_FAILED. Description: If you're seeing this message after snoozing or dismissing a reminder and this is for a calendar you do not want to add or edit events for, you can mark this calendar as read-only to avoid such experience in future. To do so, get to the calendar properties by right-clicking on this calendar in the list in the calendar or task view.

I decided to rollback to thunderbird 102 and using a new profile, it worked fine (using app-specific password).

I then upgraded to 115.1.0 and with both this upgraded new profile, as well as another brand new profile, I received the same issue I detailed in my first reply (this is with oauth). Furthermore, I was able to get proper verbose logging at least one (doesn't seem to work after a TB restart... probably another thunderbird bug):

`
Calendar: CalDAV: New webdav-sync Token: data:,1512617967-107247 CalDavRequestHandlers.jsm:935
Calendar: aChangeLogListener=[xpconnect wrapped calIGenericOperationListener]
calendarURI=https://caldav.fastmail.com/dav/calendars/user/[e-mail]/unique-id-1-goes-here/ iscached=true this.mQueuedQueries.length=0 CalDavCalendar.jsm:1060
Calendar: [calCachedCalendar] replayChangesOn finished. calCachedCalendar.js:323
Calendar: [calCachedCalendar] Performing playback operation add on 0 items to Personal calCachedCalendar.js:556
Calendar: [calCachedCalendar] Performing playback operation modify on 0 items to Personal calCachedCalendar.js:556
Calendar: [calCachedCalendar] Performing playback operation delete on 0 items to Personal calCachedCalendar.js:556
Calendar: [calCachedCalendar] Doing changelog based sync for calendar https://caldav.fastmail.com/dav/calendars/user/[e-mail]/unique-id-goes-here/ calCachedCalendar.js:304
Calendar: CalDAV: send (PROPFIND https://caldav.fastmail.com/dav/calendars/user/email%40email.com/unique-id-goes-here/): <?xml version="1.0" encoding="UTF-8"?>
<D:propfind xmlns:D='DAV:' xmlns:C='urn:ietf:params:xml:ns:caldav' xmlns:CS='http://calendarserver.org/ns/'><D:prop><D:resourcetype/><D:owner/><D:current-user-principal/><D:current-user-privilege-set/><D:supported-report-set/><C:supported-calendar-component-set/><CS:getctag/></D:prop></D:propfind> CalDavRequest.jsm:126

Calendar: CalDAV: recv: No session found via bearer token CalDavRequest.jsm:145
Calendar: CalDAV: Status 401 on initial PROPFIND for calendar Personal CalDavCalendar.jsm:1517

[calCachedCalendar] replay action failed: <unknown>, uri=https://caldav.fastmail.com/dav/calendars/user/[e-mail]/unique-id-goes-here/, result=2147500037, operation=[xpconnect wrapped calIOperation]
`

Comment 7

[Rob Mueller]

So I'm one of the developers at Fastmail.

A little while back I helped get OAuth2 support into TB for Fastmail (https://bugzilla.mozilla.org/show_bug.cgi?id=1785240). At the time I believed everything was working, but we explicitly left OAuth2 disabled. As I described:

I was able to get the entire setup process working for mail, contacts and calendars on my Fastmail testbed using these changes. However they won't currently work on Fastmail production because the Fastmail mozilla autodiscovery is returning <authentication>password-cleartext</authentication> rather than <authentication>OAuth2</authentication>. We did try returning both a while back based on the documentation at https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat#OAuth2 which says:
"Note that there are two <authentication> elements. This allows a fallback, in case a client does not support OAuth2 or does not have a client key for this OAuth2 issuer and therefore cannot authenticate with this issuer. "
We tried enabling that a while back, but it does not work correctly and causes errors to be displayed to users, so we had to disable it again.

So because the fallback didn't work, we couldn't enable it in the autodiscovery file for each domain until the code was actually shipped to a majority of people.

That was all fine and we left things as they were. However what I didn't fully appreciate is that enablement of OAuth2 for caldav/carddav is completely independent of the autodiscovery file. The autodiscovery for contacts/calendars is all done internally, and because it now autodiscovers carddav.fastmail.com as the contacts domain, and it has that domain hard-coded in the TB code base (since the 115 release) as a "does OAuth2" domain internally, it tries to initiate the OAuth2 flow for address books, but only for address books.

This created a horrible and messy flow. You enter an email address, and it goes:

1. For email, it finds the domains Autoconfiguration XML file. Since that says to use "password-cleartext" authentication, TB needs a password and prompts for that. But the password the user would need to enter here is an app specific password with IMAP/SMTP access that they would have to have configured in their Fastmail account first
2. For contacts, it uses rfc6764 to discover the carddav domain as carddav.fastmail.com, which is hard coded as using OAuth2, so TB opens up a browser window to initiate the OAuth2 login flow, which shows the Fastmail login screen where they enter their regular username + password + any 2FA credentials if required
3. For calendars, something similar to (2) is happening, but something seems messed up

This is obviously a terrible and confusing user experience.

At that point we decided the best path forward was to actually just to enable OAuth2 in the Autoconfiguration XML file for all domains for
email discovery as well. We did that, and that at least fixed the mess of multiple password boxes that would pop up. Instead everything now just attempts the OAuth2 login flow. Once the user completes that, it appears that email syncing started to "just work". Great.

However it seems that contacts and calendars are both still broken. Unfortunately I don't know exactly why this is. In my original testing I'm sure I got contacts and calendars working, but since then something broke them. I'm going to build a recent TB from scratch again and see if I can reproduce and work out what's going on in the code that's causing this.

Comment 8

[Wayne Mery (:wsmwk)]

Thanks, Rob, for your helpful comments.

(In reply to Rob Mueller from comment #7)

...
At that point we decided the best path forward was to actually just to enable OAuth2 in the Autoconfiguration XML file for all domains for
email discovery as well. We did that, and that at least fixed the mess of multiple password boxes that would pop up. Instead everything now just attempts the OAuth2 login flow. Once the user completes that, it appears that email syncing started to "just work". Great.

However it seems that contacts and calendars are both still broken. Unfortunately I don't know exactly why this is. In my original testing I'm sure I got contacts and calendars working, but since then something broke them. I'm going to build a recent TB from scratch again and see if I can reproduce and work out what's going on in the code that's causing this.

See https://www.reddit.com/r/Thunderbird/comments/16j250v/115_update_network_calendar_no_longer_works/

What's next?

Comment 9

[Rob Mueller]

So I found why the addressbook syncing wasn't working. Basically by default at Fastmail if you request a new oauth access token from a refresh token, we return an access token and a new updated refresh token as well. This is all within the oauth spec, and works fine for most systems if they have a single locked consistent token store. That's not the case in thunderbird though, where each subsystem requests it's own access token on demand and doesn't assume it will get an updated refresh token that it needs to store back.

Fortunately we have a flag we can set on each oauth client to enable/disable this feature. I disabled it for thunderbird, so now there's only a single refresh token that's not updated on each request for a new access token. With that in place, syncing for mail and addressbooks just started working.

However calendars is still broken, and I don't really understand why. Thunderbird does request an oauth token with all the right scopes (e.g. mail, carddav and caldav), but then doesn't seem to attempt to use the oauth token for calendar syncing, and I don't know why. Again, I thought I tested all this originally when I wrote the patches that were accepted, but it definitely isn't working in the most recent TB.

This requires a bunch of digging through and adding logging to code to understand how thunderbird is using oauth tokens in the calendar part of the code, and I just haven't had time for that recently. If there's someone at mozilla with experience with the calendar authentication code that could help with this, I'd love to work with them to track down what's happening here.

Comment 10

[Tom Keddie]

Any there any updates to this bug? Thunderbird 115 has rolled out onto ubuntu recently it seems and all my calendars have this problem.

Comment 11

[rb]

There appears to be a workaround where if you re-add the calendar then it will allow you to resume using app-passwords and network sync is back to normal for the new instance of it (when using Fastmail calendars).

If you've added entries to the local copy of your calendar after the sync stopped working then those entries would not carry over.automatically. But you may be able to edit and re-assign them from the old instance of the calendar to the newly created one.

After anything that needs to be copied over is taken care of you could disable (or delete) the old instance of the calendar with the broken sync/login.

Comment 12

[Wayne Mery (:wsmwk)]

Sean, having been blessed with oauth fun in the past, can you offer advice regarding comment 9?

Comment 13

[Wayne Mery (:wsmwk)]

Is this still seen in version 128?

Comment 14

[Rob Mueller]

(In reply to Wayne Mery (:wsmwk) from comment #13)

Is this still seen in version 128?

I believe this is still not working. Again... "this requires a bunch of digging through and adding logging to code to understand how thunderbird is using oauth tokens in the calendar part of the code, and I just haven't had time for that recently. If there's someone at mozilla with experience with the calendar authentication code that could help with this, I'd love to work with them to track down what's happening here."