1429644 - trying to override for Mail protocol if domain has HSTS enabled, no appropriate feedback in Thunderbird

Status: NEW

(Thunderbird :: Security, defect, P3)

Description

[SeriousSans]

Attachment: thunderbird-issue-error-dialog-2018-01-10.jpg

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Steps to reproduce:

With clean Thunderbird install and no existing ~/.thunderbird directory: 

* Thunderbird is run
* I choose to add a new mail account
* I enter the IMAP server information
* Thunderbird automatically finds the server information properly (username must be edited, however)
* When the 'Done' button is clicked, the 'Add Security Exception' dialog displays because our mail server previously used a self-signed certificate.  It now uses a CA-signed cert but it doesn't include the 'mail' domain.
* With a check mark in 'Permanently store this exception', I click 'Confirm Security Exception'

- - -

Client: Thunderbird Linux 64-bit 52.5.2 Release
OS: Debian Linux Testing (Buster/Sid) - AMD 64


Actual results:

After 'Confirm Security Exception' is pressed, the 'Add Security Exception' dialog immediately displays again, with the same information as the previous dialog.  Pressing the 'Confirm Security Exception' momentarily dismisses the dialog, only for it to come up again and again.

The error console output for this:
- - - -
mail.mydomain.com:143 uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.
The server might not be sending the appropriate intermediate certificates.
An additional root certificate may need to be imported.
The certificate is only valid for the following names:
  mydomain.com, www.domain.com  

Error code: <a id="errorCode" title="SEC_ERROR_UNKNOWN_ISSUER">SEC_ERROR_UNKNOWN_ISSUER</a>
- - - -

(I already know the certificate was self-signed or it didn't include the 'mail' domain and expect only one warning)

The account creation process never completes and mail is never downloaded.

I have even tried setting the trust for the certificate in Thunderbird so it will be trusted no matter what, but this does not change the problem.

I have also tried using the 'Advanced config' button to enter server information manually, but this does not change the problem.


Expected results:

Once I clicked the 'Confirm Security Exception' button and the dialog is dismissed, I expected the new mail account to be created and should not have seen the 'Add Security Exception' dialog repeatedly display.

Comment 1

[SeriousSans]

Attachment: thunderbird-issue-cert-view-2018-01-10.jpg

Comment 2

[Jorg K (CEST = GMT+2)]

Sounds a bit like bug 1363838.

Comment 3

[SeriousSans]

(In reply to Jorg K (GMT+1) from comment #2)
> Sounds a bit like bug 1363838.

It sounds a little bit like it, but I don't have kill Thunderbird, I can just click 'Cancel'.  I can't even get passed the 'Add Security Confirmation' dialogs and the account is never set up (unless 'Manual config' is clicked, but the problem still persists when I try to check mail).

Comment 4

[James Rome]

Attachment: tbexception.png

I get this dialog because my shared mail host uses its own certificate. I cannot get out of this except by closing TB. It comes up I think from the smtp server configuration. Pressing confirm exception or cancel just brings up the window again.

Comment 5

[SeriousSans]

(In reply to James Rome from comment #4)
> Created attachment 8948522 [details]
> tbexception.png
> 
> I get this dialog because my shared mail host uses its own certificate. I
> cannot get out of this except by closing TB. It comes up I think from the
> smtp server configuration. Pressing confirm exception or cancel just brings
> up the window again.

Thanks James.

This sounds like the exact same thing mine was doing.  This is a real deal-breaker Mozilla!  How can folks use Thunderbird if they can't create an exception for self-signed certificates?  

I had to go so far as to create a Let's Encrypt cert for my mail server, then Thunderbird would behave itself...but this is not correct behavior.  Geary, Balsa, Claws Mail, Sylpheed and others will store the self-signed cert exception just fine.  Very frustrating for Thunderbird end users.

Comment 6

[Tristan Miller]

(In reply to James Rome from comment #4)
> I cannot get out of this except by closing TB. It comes up I think from the
> smtp server configuration. Pressing confirm exception or cancel just brings
> up the window again.

Same here with Thunderbird 52.9.1 (64-bit) on openSUSE Tumbleweed.  Presumably due to some misconfiguration (but maybe due to some MITM attack, for all I know), one of the IMAP servers I use occasionally sends out an invalid certificate.  This causes the "Add Security Exception" dialog to pop up.  Clicking on "Cancel" simply causes the dialog to immediately reappear.  This makes Thunderbird completely unusable.  I can't even shut down Thunderbird other than by killing it from the command line.

Comment 7

[nu.vivo+github]

Hello,

Same loop windows her, our configuration:
* let's encrypt SSL
* Thunderbird 60.2.1 (Windows 32Bit)
* on Windows 10

Thunderbird don't get FullChain Certificate and think it's not signed by an authority

> openssl s_client -showcerts -connect <mymx>:993

Get all chains of validition.

Comment 8

[Rob Kampen]

Just had the same error - running on CentOS 7 - fully up to date - standard TB 60.6.1 (64-bit)
Mail account is hosted by nextmeta and they use a wildcard via certbot i.e. *.nextmeta.com which I can see via "view certificate"
Our domain has a mx record that points to mail.<our domain> which in turn resolves to a nextmeta.com mail server IP address.
So I understand why TB chokes on the initial presentation of the certificate, what I do not get it why it locks into an endless loop that requires me to kill the linux process in order to do anything further with TB.

Looks like this problem has been around for a while .... anyone interested in working on this??
I recognise this is an edge case and there are better ways of setting it up, but having TB lock up and forcing me to either remove the account, or to re-arrange my DNS records seems rather arrogant. If I am prepared to accept this, TB should allow me to load an exception.

Seems FF amd TB are getting more intransigent with their design / implementation. I have already relegated FF to use for testing of web sites I design and run only, as daily use was getting untenable, maybe time to go look for another mail client too. Seems a shame as I have used both since they came onto the scene some 15 years ago and pointed many folk to these once excellent tools.
Thanks for reading, I'm not holding my breath.
Rob

Comment 9

[Kai Engert [:KaiE:]]

Thanks to James for including information about the mail server, which allowed me to reproduce and confirm the issue. I will debug what's going on.

Comment 10

[Kai Engert [:KaiE:]]

(In reply to James Rome from comment #4)

It comes up I think from the
smtp server configuration.

That's very helpful.

The shared Mozilla SSL/TLS code requires a separate security override for each individual port on which an invalid certificate is being used.
I see that an override for the IMAP port (143) was added, but none for the SMTP port, which causes the code to get another (internal) error report. And then the TB might mistakenly conclude it's still about the IMAP port.

Comment 11

[Kai Engert [:KaiE:]]

After analyzing the issue, I understand what's happening, at least for the server that James had reported (as seen in the screenshot).

You experience a side effect of domain configuration per RFC 6797, and code reused by Thunderbird that is original Firefox code and is hardcoded to assume that all SSL/TLS communication is for the HTTPS protocol.

RFC 6797 defines a flag, that a web server may set, called HTTP Strict Transport Security (HSTS).
If the domain administrators sets the flag, it means "Don't allow users to override any bad certificate errors.", as explained in section 12.1 of that RFC.

For James' domain, the flag is set, and consequently the core TLS code ignores any override that Thunderbird attempts to set.

What does this mean for this bug?

First, if you are an operator of services, it could be a reasonable expectation that you need to be consistent with your operation of services. If you decided by yourself that you want to raise the security level of your primary domain to set the HSTS flag, then you shouldn't operate any services that use mismatching certificates. Or, if you really need to operate services with mismatching certificates, the obvious workaround is to relax the configuration of the webserver on your primary domain.

Whether or not HSTS is enabled for your domain can be checked with online services such as https://www.ssllabs.com/ssltest/

Second, regarding Thunderbird's user interface, it is obviously confusing. But it was written at a time when overrides always worked. The Thunderbird code would need to be enhanced to report that overriding is impossible because of domain configuration.

Third, we could debate if mail protocols should be excluded from the HSTS domain configuration. If yes, we'd need to enhance the Firefox code to allow Thunderbird to override the HSTS constraints, whenever Thunderbird operates using a non-HTTP protocol. Or alternatively, Thunderbird would have to patch the common code to disable the HSTS checks.

Comment 12

[Kai Engert [:KaiE:]]

Just as an additional explanation what happens in detail:

For example.com, Thunderbird obtains the hostname for mail, e.g. imap.example.com

Then the common SSL/TLS code attempts to check if a HSTS configuration is available, by attempting a connection to https://imap.example.com

In my test scenario, that connection failed. Consequently, the code walked up one level of the domain, and connected to https://example.com

That worked, and the server has HSTS enabled. Consequently, all subdomains of example.com are treated as "HSTS is enabled".

Comment 14

[gene smith]

Attachment: hsts-relax-mozilla.diff

I tend to agree with Kai's last paragraph of comment 11 (or at least it's the easiest way to fix the problem). Anyhow, since I can find no mention of HSTS used with imap or smtp in RFCs (just http/https), I would vote for allowing the override when TB is running the "common" mozilla code. I use an existing pref in the patch to do this; it was originally a bool and I changed it to an int so the 0 = disable the preload list, 1 = allow the preload list and 2 = thunderbird is running so skip the HSTS restrictions. When mozilla (not tb) is running only the values 0 and 1 will occur and HSTS will have its normal effect.

A known hole in this approach is when the tb "startup page" is running https with HSTS restrictions.

I'm attaching a possible diff for the mozilla and a smaller diff for comm code. Probably a hack but it does seems to work, at least for the example url denydarko.tk which emits the HSTS header and is also, in the trunk, is in the preload list and has a url mismatch certificate error that must be allowed before password prompt occurs.

Comment 15

[gene smith]

Attachment: hsts-relax-comm.diff

Comment 16

[Deny]

Let's note that this only happens for a subdomain (imap.domain.com for example) if the IncludeSubDomains detective is enabled on the HSTS header:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security#Directives

While (I assume) this is mostly useful for www, other subdomains such as IMAP or SMTP are also included.

Comment 17

[gene smith]

In addition to the start-up page, more holes is when tb displays remote https links in html email and maybe when suite/seamonkey is built and used (which I have no experience with its build process or usage).

(In reply to MTU from comment #16)

While (I assume) this is mostly useful for www, other subdomains such as IMAP or SMTP are also included.

I guess the question is whether tb's supported messaging protocols (imap, smtp, pop3, nntp etc) should respect the HSTS restrictions when they are designed, it seems, only for http[s] access?

Currently tb has no knowledge or awareness that an HSTS problem has occurred or if a mail host supports it. Somehow that would have to be communicated back to tb and when the issue occurs display a new "impossible to connect due to HSTS problem" dialog. (There is an error code that tb sees when the tcp/ip connection terminates unexpectedly due to certificate problems but it doesn't indicate that HSTS is in effect.)

Comment 18

[Deny]

(In reply to gene smith from comment #17)

In addition to the start-up page, more holes is when tb displays remote https links in html email and maybe when suite/seamonkey is built and used (which I have no experience with its build process or usage).

(In reply to MTU from comment #16)

While (I assume) this is mostly useful for www, other subdomains such as IMAP or SMTP are also included.

I guess the question is whether tb's supported messaging protocols (imap, smtp, pop3, nntp etc) should respect the HSTS restrictions when they are designed, it seems, only for http[s] access?

Currently tb has no knowledge or awareness that an HSTS problem has occurred or if a mail host supports it. Somehow that would have to be communicated back to tb and when the issue occurs display a new "impossible to connect due to HSTS problem" dialog. (There is an error code that tb sees when the tcp/ip connection terminates unexpectedly due to certificate problems but it doesn't indicate that HSTS is in effect.)

Well, frommy understanding, while TB doesn't know why, something common in TB and FF does and fails it.

Can't TB access this method itself and get the result?

Comment 19

[gene smith]

AFAIK, tb and ff don't run from the same same shared libary or dll file; they are completely separate programs. However they both build from the "mozilla" code base which handles the low level network access. But I don't really have the experience or knowledge of how they interact to answer your question. But, also AFAIK, there is no interface now to accomplish what you ask.

Comment 22

[gene smith]

Attachment: hsts-relax-mozilla-v2.diff

The previous diff (hsts-relax-mozilla.diff) just skipped checking and acting on the HSTS host list for thunderbird. Now for thunderbird the list is checked and if host certificate has a problem it will now only allow override if the port to connect to is 143 and 993. So tb will still disconnect and not override when accessing strict sites with bad certs using non-imap methods, like URLs in emails. (This does not address the poor error reporting if this happens.)

This probably should also allow the ports to send mail via the well-know SMTP ports and receiving via well know POP3 and NNTP ports too. With this patch, those protocols will be rejected if the server is on the strict list and the cert is bad.

Note: For testing I added tana.it to the the "strict list": security/manager/ssl/nsSTSPreloadList.inc. This is only for testing. This allowed a duplication of the problem described in Bug 1690983 which is fixed by this patch.

Comment 23

[gene smith]

Attachment: hsts-relax-comm-v2.diff

This is the proposed patch for the comm-central code. This causes the mozilla code to see the pref value as 2 meaning "this is thunderbird".

Comment 24

[Magnus Melin [:mkmelin]]

I'm adjusting the title, since there's a way to override, per bug 1690983, just not very discoverable:
network.stricttransportsecurity.preloadlist false + security.cert_pinning.enforcement_level 0.

I don't think we should pursue making it easier to override if the domain uses HSTS. Imagine how much more easy it would be to attack e.g. our gmail users. One click at a cafe and you're hosed. There's a multitude of ways to fake SSL in such a way that users would get tricked into accepting such an override.

The only thing left is to give better error message so that super informed users can set the prefs if they need to.

Comment 25

[Ricarda Weber]

@Magnus, I fully agree with you. An informed user can override, that is fully sufficient. It need not be made easier than it is already.

But please come up with an appropriate and clear error message: "The Domain <xyz.ab> is under HSTS and the server <fgh.xyz.ab> is not providing a proper certificate (<view certificate>). You cannot continue at this point.".

Just a "grant a permantent exception" dialog an/or dropping TLS without comment is not helpful. This leads to a lot of waste of time when trying to figure out the cause of the problem.

Comment 26

[gene smith]

Magnus in comment 24 said:

I don't think we should pursue making it easier to override if the domain uses HSTS. Imagine how much more easy it would be to attack e.g. our gmail users. One click at a cafe and you're hosed. There's a multitude of ways to fake SSL in such a way that users would get tricked into accepting such an override.

Ok, I see your point.

So when tb does a connection to a imap.gmail.com account using TLS, maybe at a cafe, and imap.gmail.com is resolved to a fake address that returns a bad certificate, what happens? Tb puts up the override dialog for the certificate. Since gmail.com is on the "preload" list, the override won't work. So I think the proposal is to put up the message "The Domain gmail.com is under HSTS and the server imap.gmail.com is not providing a proper certificate (<view certificate>). You cannot continue at this point." instead of the override message.

So once this new message appears, how does a "super informed user" who is sure the certificate is really OK know to set preference network.stricttransportsecurity.preloadlist to false? Maybe google and stumble on this bug? Or should there be something in the proposed message about this pref?

Once it is set to false, should there be some warning or maybe a special icon to indicate that tb is running in a somewhat less secure state?

Removing myself as "assignee" for now and obsoleting my patch diffs since the preferred solution seems to be mostly in the UI realm.

---

Wrote the above a few days ago and did a bit more research. It seems with the tb current code it is never possible to override a bad gmail certificate in tb, at least with the parameters we've been discussing. This is probably because of the OAUTH2 authentication. Even with network.stricttransportsecurity.preloadlist set false at this point in the process entry is null so override can't complete:
https://searchfox.org/mozilla-central/rev/b32d4ca055ca9cf717be480df640f8970724a0ce/security/manager/ssl/nsCertOverrideService.cpp#563
FYI, I caused tb to reject gmails certificate by setting gmail server name to one of their internal servers: yx-in-f108.1e100.net

Comment 27

[Magnus Melin [:mkmelin]]

Not sure how exact error code we get back. These places https://searchfox.org/comm-central/search?q=Ci.nsINSSErrorsService.ERROR_CLASS_BAD_CERT&path= should probably be extended to give feedback that "The cert was bad. Error code: x". or perhaps a bit more fancy. See bug 1681960 as well.