Closed Bug 1769493 Opened 3 months ago Closed 16 days ago

Thunderbird auto detection of CalDAV and CardDAV via RFC6764 isn't working

Categories

(Calendar :: General, defect)

Thunderbird 91
defect

Tracking

(thunderbird_esr91 wontfix, thunderbird_esr102+ affected)

RESOLVED FIXED
104 Branch
Tracking Status
thunderbird_esr91 --- wontfix
thunderbird_esr102 + affected

People

(Reporter: Mikeyy, Assigned: mkmelin)

Details

(Whiteboard: [TM:102.1.2])

Attachments

(1 file)

Steps to reproduce:

All testing was done with PortableApps Thunderbird 91.9, Windows 10 64bit, on my email domain miho.im
Couldn't use the download from thunderbird.net since I have an active business installation.

SRV and TXT records were set according to RFC6764 specification. You can check them here: https://network-tools.com/nslookup/
Just enter the following into the search field and pick SRV or TXT records.
_caldavs._tcp.miho.im
_carddavs._tcp.miho.im

You will see that SRV records are pointing to miho--im.w.emailarray.com and TXT records are defining the additional path to /caldav and /carddav, so in total:
miho--im.w.emailarray.com/caldav
miho--im.w.emailarray.com/carddav

As a backup, there is also .well-known URL active which is also part of RFC6764 specification. It can be found here:
https://miho--im.w.emailarray.com/.well-known/caldav
https://miho--im.w.emailarray.com/.well-known/carddav

It's redirecting to the above-mentioned URLs.

  1. Start Thunderbird with a blank profile. The account creating screen is shown.
  2. Enter email address and password.
  3. IMAP server and protocol is correctly detected via Thunderbird auto-config file (https://wiki.mozilla.org/Thunderbird:Autoconfiguration).
  4. When detected configuration is confirmed via click on DONE button, the account setup screen switches to the calendar and address book autodetection.
  5. After some time searching for configuration (20 seconds of "Looking up address book" and 85 seconds of "Looking up calendars"), Thunderbird finished without any info. It doesn't say sucess or fail, nothing.
  6. Only option left is to press "Finish" button.

Whole time TB is autodetecting caldav and carddav, email from IMAP account is syncing in background, so username and password is correct.

According to RFC6764 TB should detect SRV record and search for TXT path after that. If TXT path is missing, it should test .well.known address on that domain.

Maybe I defined SRV and TXT records incorrectly, but it doesn't look like it according to specs.

Please set the carddav.setup.loglevel pref to All
Then go to Address book | New CardDAV address book (toolbar button)

Check the dev console. (It says found SRV and TXT records, and will try to connect. After that I can't continue as no creds.)

Deleted profile folder for portable TB and changed carddav.setup.loglevel to All.
Went to create a new account and in error console was this:
carddav.setup: Looking up DNS record for emailarray.com CardDAVUtils.jsm:288:11
carddav.setup: Attempting to connect to https://emailarray.com/.well-known/carddav CardDAVUtils.jsm:396:11
mail.setup:
Exception { name: "NS_ERROR_NET_TIMEOUT", message: "Connection failure", result: 2152398862, filename: "resource:///modules/CardDAVUtils.jsm", lineNumber: 202, columnNumber: 0, data: null, stack: "onStreamComplete@resource:///modules/CardDAVUtils.jsm:202:20\n", location: XPCWrappedNative_NoHelper }
accountSetup.js:2325
Calendar: [CalICSProvider] Could not detect calendar using method attemptHead - HTTP response status -1 CalICSProvider.jsm:101
Calendar: [CalDavProvider] Could not detect calendar using method wellKnown - HTTP response status -1 CalDavProvider.jsm:106
Calendar: [CalICSProvider] Could not detect calendar using method attemptGet - HTTP response status -1 CalICSProvider.jsm:101
Calendar: [CalDavProvider] Could not detect calendar using method attemptRoot - HTTP response status -1 CalDavProvider.jsm:106
services.settings: thunderbird/search-config has signature disabled RemoteSettingsClient.jsm:1027
services.settings: thunderbird/hijack-blocklists has signature disabled RemoteSettingsClient.jsm:1027
Calendar: [CalICSProvider] Could not detect calendar using method attemptDAVLocation - HTTP response status -1 CalICSProvider.jsm:101
Calendar: [CalICSProvider] Could not detect calendar using method attemptPut - HTTP response status -1 CalICSProvider.jsm:101
mail.setup: NoneFoundError:
DetectionError resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20
<anonymous> resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:31
detect resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:164
fetchCalendars chrome://messenger/content/accountcreation/accountSetup.js:2439
showSuccessView chrome://messenger/content/accountcreation/accountSetup.js:2296
accountSetup.js:2448
NS_ERROR_NOT_AVAILABLE

From first two line it seams that TB is searching for carddav on emailarray.com instead of defined miho--im.w.emailarray.com
Also, not sure why it didn't just straight to miho--im.w.emailarray.com/carddav as defined with SRV + TXT

Ok, when I go directly via address book it's different:

carddav.setup: Looking up DNS record for miho.im CardDAVUtils.jsm:288:11
carddav.setup: Found a DNS SRV record pointing to miho--im.w.emailarray.com CardDAVUtils.jsm:295:13
carddav.setup: Found a DNS TXT record pointing to https://miho--im.w.emailarray.com/carddav CardDAVUtils.jsm:305:15
carddav.setup: Attempting to connect to https://miho--im.w.emailarray.com/carddav CardDAVUtils.jsm:396:11
carddav.setup: https://miho--im.w.emailarray.com/carddav ... success CardDAVUtils.jsm:399:13
carddav.setup: Attempting to connect to https://miho--im.w.emailarray.com/carddav/principals/xyz@miho.im/ CardDAVUtils.jsm:396:11
carddav.setup: https://miho--im.w.emailarray.com/carddav/principals/xyz@miho.im/ ... success CardDAVUtils.jsm:399:13
carddav.setup: Attempting to connect to https://miho--im.w.emailarray.com/carddav/addressbooks/xyz@miho.im/ CardDAVUtils.jsm:396:11
carddav.setup: https://miho--im.w.emailarray.com/carddav/addressbooks/xyz@miho.im/ ... success CardDAVUtils.jsm:399:13
carddav.setup: Saving login info for xyz@miho.im CardDAVUtils.jsm:558:17

Somehow DNS detection is different when accessing it via the account creation screen.

Component: Untriaged → General
Product: Thunderbird → Calendar
Assignee: nobody → mkmelin+mozilla
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Thanks for debugging!

Target Milestone: --- → 102 Branch

Hi, I tested this feature with the newest Daily 102.0a1 (2022-05-23) - it seams that the DNS-SRV/TXT resolving now works - but for us then fails on authorization:

carddav.setup: Looking up DNS record for [Mailserver-Domain] CardDAVUtils.jsm:289:11
carddav.setup: Found a DNS SRV record pointing to [CalDAV/CardDAV Server Host] CardDAVUtils.jsm:296:13
carddav.setup: Found a DNS TXT record pointing to [CalDAV/CardDAV Server URL] CardDAVUtils.jsm:306:15
carddav.setup: Attempting to connect to [CalDAV/CardDAV Server URL]

mail.setup:
Exception { name: "NS_ERROR_FAILURE", message: "Authorization failure", result: 2147500037, filename: "resource:///modules/CardDAVUtils.jsm", lineNumber: 209, columnNumber: 0, data: null, stack: "onStreamComplete@resource:///modules/CardDAVUtils.jsm:209:15\n", location: XPCWrappedNative_NoHelper }
accountSetup.js:2458

Calendar: [CalDavProvider] Could not detect calendar using method dnsSRV - resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20: AuthFailedError - DetectionError@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20:1
@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:31:7
detectCollection@resource:///modules/CalDavProvider.jsm:285:13

Calendar: [CalICSProvider] Could not detect calendar using method attemptDAVLocation - resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20: AuthFailedError - DetectionError@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20:1
@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:31:7
attemptDAVLocation@resource:///modules/CalICSProvider.jsm:250:13
CalICSProvider.jsm:81

mail.setup: AuthFailedError:
DetectionError resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:20
<anonymous> resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:31
detectCollection resource:///modules/CalDavProvider.jsm:285
accountSetup.js:2581

-> Using the "normal" setup in the calendar and address tabs works fine it is just the initial account creation

A hint may be, that our username for both IMAP/SMTP and CalDAV/CardDAV is the same - but it is not the E-Mail address (for IMAP/SMTP configured by autoconfig server)

I don't think this patch is part of TB nightly, at least not yet.

Correct. I need to adjust a test before it can land.

Hi Magnus,
would it be possible to add for that general account setup also the option if CalDAV/CardDAV DNS lookup fails, to check also maildomain or insert serverurl manually - like its offered if user trys to add it in addressbook or calendar?

EGroupware is (among others) an open source server for caldav/carddav. We support autodetection with ".wellknown" and inserting serverurl detects the user home-set correctly. But customers often have to less knowledge of the dns and can't manage to set it correctly. Additionally in our EGroupware cloud we use wildcard certificates where dns can't be set for this kind of things. So for us the option to insert server url would be really helpful, if DNS is not set. It would help that people don't need to configure it in the next step manually for CalDAV and CardDAV and insert there again username, password and serverurl (same for CalDAV and CardDAV).

Kind regards
Birgit Becker
EGroupware (https://github.com/EGroupware/egroupware)

Target Milestone: 102 Branch → ---

This was solved 2 months ago. Any chance of fix hitting 102?

(In reply to Mihovil Stanic [:Mikeyy - L10n HR] from comment #10)

This was solved 2 months ago. Any chance of fix hitting 102?

There has been no code change associated with this bug report in Thunderbird yet. What solved your problem then?

Flags: needinfo?(mihovil)

Obviously, my problem isn't solved since it hasn't reached release versions.
It's normal to assume that since Magnus found the root of the problem (using server domain instead of email domain), and autodetection in the address book and calendar is working without issues, his fix has a good chance of working from the first try.

Flags: needinfo?(mihovil)

I haven't had time to finish the patch yet - it is at least failing a test. Hoping to get back to it soon.

Attachment #9276893 - Attachment description: Bug 1769493 - Make CalDAV/CardDAV detection during account setup start off with email domain, not domain of detected incoming server. r=darktrojan → Bug 1769493 - Make CalDAV/CardDAV detection during account setup start off with email domain, not domain of detected incoming server. r=aleca

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/e482bbf11ffd
Make CalDAV/CardDAV detection during account setup start off with email domain, not domain of detected incoming server. r=aleca

Status: ASSIGNED → RESOLVED
Closed: 16 days ago
Resolution: --- → FIXED

Mihovil: please try today's daily.

Target Milestone: --- → 104 Branch

Thank you, it works!

I have an issue with GUI since it's not adding address books or calendars by default, you need to click on each, but they are hidden by default and you are only offered the "Finish" button.
But I'll report that in a separate bug.

For me it still fails. Mail is configured correctly, but no CalDAV CardDAV setup happens. SRV-Records are correct (manual configuration with URL and .well-known works perfectly fine).

As a tiny regression to last Time above, it didn't try looking up the CalDAV-Path via TXT anymore, but went directly to .well-known

Thunderbrid version: 104.0a1 (2022-07-23) (64-Bit)

carddav.setup: Looking up DNS record for example.org CardDAVUtils.jsm:291:11
carddav.setup: Found a DNS SRV record pointing to servername.example.org CardDAVUtils.jsm:298:13
carddav.setup: Attempting to connect to https://servername.example.org/.well-known/carddav CardDAVUtils.jsm:399:11

mail.setup:
Exception { name: "NS_ERROR_FAILURE", message: "Authorization failure", result: 2147500037, filename: "resource:///modules/CardDAVUtils.jsm", lineNumber: 211, columnNumber: 0, data: null, stack: "onStreamComplete@resource:///modules/CardDAVUtils.jsm:211:15\n", location: XPCWrappedNative_NoHelper }
accountSetup.js:2466
Calendar: [CalICSProvider] Could not detect calendar using method attemptDAVLocation - resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:19: AuthFailedError - DetectionError@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:19:1
@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:30:7
attemptDAVLocation@resource:///modules/CalICSProvider.jsm:249:13
CalICSProvider.jsm:80
Calendar: [CalDavProvider] Could not detect calendar using method dnsSRV - resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:19: AuthFailedError - DetectionError@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:19:1
@resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:30:7
detectCollection@resource:///modules/CalDavProvider.jsm:283:13
CalDavProvider.jsm:84
mail.setup: AuthFailedError:
DetectionError resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:19
<anonymous> resource:///modules/calendar/utils/calProviderDetectionUtils.jsm:30
detectCollection resource:///modules/CalDavProvider.jsm:283
accountSetup.js:2589

My tests were done on two different domains.

1st domain
SRV record _carddavs._tcp.example.com + well-known url

2nd domain
SRV record _carddavs._tcp.example.com + TXT record _carddavs._tcp.example.com pointing to a subdirectory of a domain

On both domains, CardDAV address books and CalDAV calendars were detected without issues.

Hm, then it maybe really sth. with my idea in the last post, that our username for IMAP/SMTP and CalDAV/CardDAV is not the mail address (but universal unix/ldap account)?

Configuring by hand works perfectly fine and as the log show the URL gets also detected correctly (with .well-known but that also works fine), but then fails...

Comment on attachment 9276893 [details]
Bug 1769493 - Make CalDAV/CardDAV detection during account setup start off with email domain, not domain of detected incoming server. r=aleca

[Approval Request Comment]
Regression caused by (bug #): never worked
User impact if declined: may not be able to autodetect caldav/carddav in the initial setup flow
Testing completed (on c-c, etc.): beta
Risk to taking this patch (and alternatives if risky): not too risky per se, but we could keep it on beta for some time

Attachment #9276893 - Flags: approval-comm-esr102?
Whiteboard: [TM:102.1.2]
You need to log in before you can comment on or make changes to this bug.