1619135 - [autoconfig] autoconfig server on uberspace.de domain fails, because foo.uberspace.de is a public domain suffix like .co.uk

Status: RESOLVED INVALID

(Core Graveyard :: Networking: Domain Lists, defect)

Description

[Justus Winter]

Attachment: thunderbird.all.log

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

I'm trying to improve the account creation for a mail server setup of
mine. To this end, I setup and configured automx2 on the server, and
now I'm trying to get Thunderbird to use it. automx2 implements
Mozilla's autoconfiguration protocol:

https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat

This bug is also tracked in Debian, the maintainer asked me to report this upstream: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952853

Actual results:

Sadly, Thunderbird's account creation wizard fails to use this
information. I increased the log level on
mail.wizard.logging.{console,dump} to "All", and captured a log of
this interaction (attached).

The curious thing is that the autoconfiguration actually seems to
succeed:

2020-03-01 10:37:43 mail.setup INFO found config:
Incoming: imap, harrington.uberspace.de:993, SSL, auth: plain, username: (redacted), password: not set
Outgoing: smtp, harrington.uberspace.de:587, STARTTLS, auth: plain, username: (redacted), password: not set
2
2020-03-01 10:37:43 mail.setup INFO Cannot contact server
2

This is the right information, I don't understand why it says "Cannot
contact server":

% socat - tcp:harrington.uberspace.de:587
220 harrington.uberspace.de ESMTP
^C
% socat - openssl:harrington.uberspace.de:993

• OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready.
  ^C

This information is coming from the first source that Thunderbird
tries:

2020-03-01 10:37:42 mail.setup INFO Requesting https://autoconfig.probier.email/mail/config-v1.1.xml
2

% curl https://autoconfig.probier.email/mail/config-v1.1.xml?emailaddress=[e-mail]
<clientConfig version="1.1"><emailProvider id="automx2-23425"><identity /><domain>probier.email</domain><displayName>probier.email</displayName><displayShortName>probier.email</displayShortName><incomingServer type="imap"><hostname>harrington.uberspace.de</hostname><port>993</port><socketType>SSL</socketType><username>%EMAILADDRESS%</username><authentication>plain</authentication></incomingServer><outgoingServer type="smtp"><hostname>harrington.uberspace.de</hostname><port>587</port><socketType>STARTTLS</socketType><username>%EMAILADDRESS%</username><authentication>plain</authentication></outgoingServer></emailProvider></clientConfig>

Thunderbird then goes on to try Microsoft's autodiscover protocol,
which should also be implemented by automx2, but that fails (haven't
looked into this deeper). Finally, it tries some heuristic, which
mistakenly uses "probier.email" as the server name (which kinda works,
modulo TLS cert common name).

Manual configuration of the account works fine.

Expected results:

I expect autoconfiguration to work using Mozilla's own protocol. If
anything goes wrong with that (maybe automx2 doesn't correctly
implement the protocol?), I'd like to see a more informative error
message in the log.

Comment 1

[Onno Ekker [:nONoNonO UTC+1]]

Have you verified that you can connect to the IMAP-server from the same workstation as where Thunderbird is running?
From my workstation I can connect to smtp:harrington.uberspace.de:587, but I can't connect to imap:harrington.uberspace.de:993.

% curl smtp://harrington.uberspace.de:587
214 netqmail home page: http://qmail.org/netqmail

% curl imap://harrington.uberspace.de:993
curl: (56) response reading failed

Comment 2

[Philipp]

Hello, using Uberspace.de myself, I can confirm this bug.

From what I can tell the problem comes from the exception and incorrect handling thereof.

Calling Services.eTLD.getBaseDomainFromHost("harrington.uberspace.de"); results in the exception, because *.uberspace.de is on the Public Suffix List.
"Exception { name: "NS_ERROR_INSUFFICIENT_DOMAIN_LEVELS", message: "Component returned failure code: 0x804b0050 (NS_ERROR_INSUFFICIENT_DOMAIN_LEVELS) [nsIEffectiveTLDService.getBaseDomainFromHost]".

Manually using harrington.uberspace.de works without a problem. Changing automx2 (autoconfig) to use any other domain pointing to the Uberspace host works as well, unfortunately no login is possible because only harrington.uberspace.de is allowed..

I hope this can get resolved.

Side note:
Isn't it kind of pointless to request and check all other autoconfig/autodiscover methods when the first one has already returned a valid config? Especially leaking the domain to https://live.thunderbird.net/autoconfig/v1.1/ could be avoided.

Comment 3

[Ben Bucksch (:BenB)]

This bug is very specific to uberspace.de and other domains that were added to the Public Domain Suffix list.

This bug report is lacking a proper reproduction steps, and documenting what exactly happens at which stage. Where do you see the error? Where do you see the exception? I think it would be in createInBackend.js line 47.

Have you tried using imap.harrington.uberspace.de or mail.harrington.uberspace.de as hostnames? It seems that the reporter now worked around the problem, by returning "probier.email" as hostname for IMAP and SMTP, which does not fail in this way. It now fails due to lacking SSL certificates for these domains, which is a server configuartion problem, so the problem is no longer reproducable for me.

Suggesting WONTFIX.

Comment 4

[Philipp]

Ok, to help out I tried it on the current nightly build from 2021-04-13.
(Specifically on Windows with https://ftp.mozilla.org/pub/thunderbird/nightly/2021/04/2021-04-13-09-49-34-comm-central/thunderbird-89.0a1.en-US.win32.zip)

To reproduce:

1. Go into the about:config page to and set the preference "mail.setup.loglevel" to "all". (Apparently the info log level does not get reported anymore without changing this setting..)
2. Open the error console (Ctrl + Shift + J)
3. Go into the account setup and start the setup with an existing email address.
4. Name and password do not matter, but for email address enter "[e-mail]"
5. Click "Done"
6. The previously opened error console shows which automx urls are tried and also the exception(s).
   The relevant and valid configure urls to be used is "https://autoconfig.probier.email/mail/config-v1.1.xml?emailaddress=[e-mail]" (the GET param with the emailaddress is apparently not logged, but still included otherwise the call would have returned a 400 HTTP error response).
   In the included stacktrace of the shown exception you can see that it was thrown in https://searchfox.org/comm-central/source/mail/components/accountcreation/content/accountSetup.js#1124
   So from what I can tell this makes thunderbird not use this valid found configuration and go on to try other automx options which are all not correct.

I don't think having the host in the public suffix list should prevent it from being used as an email server in autoconfiguration.

Hopefully this helps.

Comment 5

[Philipp]

Ok I can't edit my post, but the paragraph about there being a 400 HTTP error is not correct in this case, please ignore it.

Comment 6

[Ben Bucksch (:BenB)]

Just FYI, harrington.uberspace.de is a domain, in this case, in the same way as yahoo.co.uk or yahoo.com. That's why this case is special. imap.harrington.uberspace.de is a hostname, in this case.

I don't think it's a particular hardship to use imap.harrington.uberspace.de instead. Given that IMAP doesn't care about the hostname, you just need to add an A record and use that in the config, and it's fixed.

accountSetup.js#1124

That's let domain = Services.eTLD.getBaseDomainFromHost(server.hostname); in _makeHostDisplayString().
This code is trying to highlight the domain, and that function simply throws, given that it's already a base domain. It's a little unfortunate API to make it throw instead of just returning the input, but that's how it is.

Even if we were to fix this, there are a number of other code places in other sections where we call the same function. It would complicate the code a number of places. I'm counting at least 5 occurances just in this dialog.

Thanks for finding the code place, but I still think it's not worth the effort, given that it affects very very few people and the trivial workaround.

WONTFIX

Comment 7

[Ben Bucksch (:BenB)]

FWIW, if we ever were to fix this, this should be fixed by Services.eTLD.getBaseDomainFromHost('yahoo.com'); returning simply yahoo.com instead of throwing an exception. That particular edge of this particular API has already annoyed me in a few cases in the past. So, it should be fixed in Gecko in any case.

Comment 8

[Philipp]

Well I have tried it out with prepending imap. and smtp. respectively.
With the nightly version it just gets stuck at "Checking password". With the current stable version 78.9.1 it works but ofc complains about the wrong SSL certificate.

Since uberspace.de is a hosting service provider I, and I believe the initial bug reporter as well, are just customers there and don't have that much control over the servers. So I will ask them if they can add the A records and issue additional certificates, but still think it shouldn't be necessary.

About fixing the bug:
Personally I think using a solution like in https://searchfox.org/comm-central/source/mozilla/browser/modules/SiteDataManager.jsm#75 and calling that instead shouldn't really complicate the code base. But then again I am not a developer here so it is highly likely that I am missing something.

I guess it was just meant as an example, but calling Services.eTLD.getBaseDomainFromHost('yahoo.com'); already returns yahoo.com. :)

Comment 9

[Ben Bucksch (:BenB)]

Interesting. I didn't know that. Indeed:

Services.eTLD.getBaseDomainFromHost("h.example.com"); // -> "example.com" - normal case
Services.eTLD.getBaseDomainFromHost("yahoo.com"); // -> "yahoo.com" - domain returns itself, as expected
Services.eTLD.getBaseDomainFromHost("h.uberspace.de"); // -> Exception { name: "NS_ERROR_INSUFFICIENT_DOMAIN_LEVELS" ... - the bug
Services.eTLD.getBaseDomainFromHost("imap.h.uberspace.de"); // -> "imap.h.uberspace.de" -- Wrong, expecting "h.uberspace.de"
Services.eTLD.getBaseDomainFromHost("h.co.uk"); // -> "h.co.uk" - works correctly. Consequently, h.uberspace.de should behave like h.co.uk

So, that really seems like a bug in the eTLD service.

Reproduction:

1. Services.eTLD.getBaseDomainFromHost("h.uberspace.de");

Actual result:

• Exception { name: "NS_ERROR_INSUFFICIENT_DOMAIN_LEVELS" ...

Expected result:

• "h.uberspace.de"

Narrowing down:

• "uberspace.de" is registered as "Public Domain Suffix", similar to "co.uk", but:
• Services.eTLD.getBaseDomainFromHost("h.co.uk"); returns "h.co.uk" as expected
• Services.eTLD.getBaseDomainFromHost("imap.h.uberspace.de"); returns "imap.h.uberspace.de", but should return "h.uberspace.de"

Comment 10

[Ben Bucksch (:BenB)]

It's a bug in the registration https://publicsuffix.org/list/public_suffix_list.dat

// Uberspace : https://uberspace.de
// Submitted by Moritz Werner <[e-mail]>
uber.space
*.uberspace.de

(this makes h.uber.space a domain, and foo.h.uberspace.de a domain, but h.uberspace.de is not even a domain, according to this registration. That's the bug.)

whereas:

co.uk

So, it should be:

uber.space
uberspace.de

Then it would work.

In other words, that really is a bug caused by the owner of uberspace.de , specifically by Moritz Werner <[e-mail]> .

Comment 11

[Philipp]

They did it like this because the individual customers have domains they control like this:

customer1.server1.uberspace.de
customer2.server1.uberspace.de

customer3.server2.uberspace.de
customer4.server2.uberspace.de

By having the entry like that they maintain separation between the individual customers, but don't have to register every new server they setup in the PSL.
For background info, all customers on a server share the mail server. So in the technical sense, server1.uberspace.de is not only a domain but also an actual server/host.

So I think the account creating component should use the eTLD service like the linked SiteDataManager.js does. Maybe there even is a more generic utils place in the code where it can be used by all components.

Comment 12

[Ben Bucksch (:BenB)]

OK, but then this bug is really invalid. If customer1.foo.uberspace.de is the customer domain, then foo.uberspace.de is really like co.uk. In fact, that's precisely what their Public Domain Suffic registration says, that harrison.uberspace.de is explicitly not a server, but on the same level as co.uk. Just as there cannot be a server or mail server with the hostname co.uk, there cannot be a mail server with the hostname harrison.uberspace.de, given the current registration.

The behavior of Thunderbird and of the Effective TLD service is completely correct here. co.uk is not a valid hostname.
So, uberspace needs to change either their server names or their Public Domain Suffix registration.
It should be trivial for them to create an imap.server.harrison.userspace.de DNS A record and use that (whereby server.harrison.userspace.de is the domain name, according to their own choice in Public Domain Suffic list).

Please note that this bug has cost me a lot of time to investigate, and it turns out to be a bug in uberspace.de, so please do not continue to argue this here, but take it up with uberspace.de, as they need to fix this on their end.

-> INVALID