Closed Bug 1783615 Opened 3 years ago Closed 2 years ago

TLS client auth no longer working in Thunderbird 102.x

Categories

(Thunderbird :: Security, defect, P2)

Product:
Thunderbird
Component:
Security
Version:
Thunderbird 102
Type:
defect

Tracking

(thunderbird_esr102 affected)

Status:
RESOLVED DUPLICATE of bug 1777717
Tracking Flags:
Tracking Status
thunderbird_esr102 --- affected

People

(Reporter: bugzilla-2022, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: regression, Whiteboard: tb-crypto-broken-feature)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0

Steps to reproduce:

Our IMAPS server uses certificate authentication, so during TLS handshake it requires the client to present a certificate. Upgrading from v91 to v102 seems to have broken this functionality. I initially ran into the problem with Debian/sid packages (v102 and v103) but I was later able to reproduce it with binaries downloaded directly from Mozilla, and I have now downgraded to Debian/stable package v91 where things work.

Actual results:

In v102 Thunderbird still asks me which certificate it should send to the server, but does not actually send it, so the server gives error 42 (did not present cert). TB seems to not even notice that the connection was closed, as the "tab is loading" animation keeps running forever, and TB does not even attempt reconnect.

Expected results:

TB should send the client cert during handshake, or renegotiation. Server side won't proceed with authentication until it sees a valid client cert.

I ran into a similar problem with Firefox v102, but I was able to fix it there by fiddling with about:config, but I couldn't find similar settings in TB's about:config. I don't remember which setting it was that fixed it, but I think it was something to do with SSL renegotiation. I think it was "security.tls.enable_post_handshake_auth"...

I wonder what has changed in Firefox core regarding TLS client auth?

Severity: -- → S2
Status: UNCONFIRMED → NEW
status-thunderbird_esr102: --- → affected
Ever confirmed: true
Keywords: regression
Priority: -- → P2
Summary: does not send certificate when requested to do so → TLS client auth no longer working in Thunderbird 102.x

Looks like maybe it got set false recently for TLS 1.3:
https://searchfox.org/mozilla-central/rev/4f2984be127d2e7c788cf1848d63dca63022beec/modules/libpref/init/StaticPrefList.yaml#12587

If security.tls.enable_post_handshake_auth isn't present in tb 102, it can be added as a boolean and set true using config editor. (It's present in my daily build and set false by default.)

Aleksi, does comment 2 help you to make it work again?

Flags: needinfo?(bugzilla-2022)

In Thunderbird, open Settings | Config Editor.

Hmm ... so even if the config editor cannot find a setting by that name, I can somehow set it to a different value?

Yes, click the plus to add it. Add it as new boolean with value true

Hm, it should be findable. It's there for me.

Blocks: tb102found

I found it, but changing the setting didn't seem to fix the issue.

Flags: needinfo?(bugzilla-2022)

I set up client certificate access to dovecot imap server using tb 91. Then I tried it with very recent daily and it worked too. It didn't require any special about:config (Config Editor) settings or added prefs to work.
With my setup, everything is self signed. I don't know if reporter Aleksi is self-signed but, if so, there is an issue with 102 in getting the override prompt to occur at all: bug 1764770
If self-signed and with the fix for bug 1764770 in place, you have to select Inbox and click "Get Messages", possibly several times, to bring up the override prompt.
If reporter Aleksi is not self-signed or if the fix from bug 1764770 doesn't help, probably need more details on his configuration and setup. Maybe even an IMAP:5 log (see https://wiki.mozilla.org/MailNews:Logging) or even a wireshark trace.

Yes, we have company internal PKI, which you can call self-signed. I don't believe I'm running into the override bug, because the root CA has already been taught to thunderbird, and besides if I reset the certificate setting it is asking me for which certificate to present. It just doesn't present it.

I tried to get an IMAP:5 log, but the files came up empty. I also tried negotiateauth:5 but that came up empty too. I'll see what I can do about the wireshark trace. I've got gigabytes of mail in my imap folders, and reseting the whole thunderbird profile will sync up all that email over the network, so I find it difficult to motivate myself to disrupt my work for this. :-(

Here's how I record the imap log from bash command line:
MOZ_LOG=IMAP:5,timestamp,sync MOZ_LOG_FILE=~/tblog ~/Downloads/thunderbird/thunderbird --allow-downgrade -p

and the log appears in ~/tblog.moz_log and you can watch it while tb is running.

If you restart tb after running with the env vars set, the log get written back to empty and then the new session is recorded. Maybe that's what's happening?
Also, the MOZ_LOG var is case sensitive, at least on linux.

Anyhow, to record the log TB just needs to see the env vars where it's running.

From comment 0:
< so the server gives error 42 (did not present cert).

What kind of server is this? Maybe dovecot like I'm testing with?

In server setting, you are set to:

Connection Security: SSL/TLS
Authentication Method: TLS Certificate

right? Just want to make sure.

Another thing to try might be to reset the certificate storage in tb. You will probably need to use the latest beta (104.0b4) or use the next official release 102.2.* when it comes out to avoid the self-signed bug. This also requires you to import the client certificate into TB so you have to have it stored where you can access it. So if you don't have it, don't do this.

Go to Privacy & Security | Certificates | Manage Certificates...
Under "Your Certificates" find and delete your client certificate
Under "Authentication Decisions" delete the same named certificate if it's there
Under "Servers" delete your (self-signed?) server certificate
Under "Authorities" find and delete only your company's authority item.

For good measure, restart tb and go back to Manage Certificates...
Under "Your Certificates" import your client certificate file (mine's called client.pfx)
Go to account Inbox and try to open a message. At this point, will have to click "Get Messages" maybe multiple times to bring up the bad certificate override dialog.
Then try to open another message and the selection for client certificate should come up. Allow tb to "remember" it and click OK.
Should be able to open any email now if this works.

I wasn't actually having any problems with my client cert login but I just did the above procedure and everything still worked as before.

Another way is to test independent of TB using openssl. I'm not an expert on this and you might already know more about this than I do, but here's how I did it.
If you already have the client certificate and it's private key as separate files you can do this:

cd <to where your client certificate is>
openssl s_client -connect <url-of-server>:993 -cert ./client.crt -key ./client.key -state -debug
:
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SPECIAL-USE LITERAL+ AUTH=PLAIN AUTH=EXTERNAL] Dovecot ready.
. authenticate external =     {you enter this, including the =}
:
* OK {CAPABILITY .... (lots more shown now that authenticated)
. select inbox                        {you enter this}
<lists stuff like EXISTS, RECENT etc, and last line has OK means it worked>

If you only have a .pfx (a.k.a., .p12) file for the client cert, you have to split out the certificate and the key from the .pfx file like this:

openssl pkcs12 -in client.pfx -out client-from-pfx.pem -nodes
openssl pkcs12 -in client.pfx -out client-key-from-pfx.pem -nodes -nocerts

Then use the output files from these with openssl s_client to connect to server as described above:

openssl s_client -connect <url-of-server>:993 -cert ./client-from-pfx.pem -key ./client-key-from-pfx.pem -state -debug

So if this works (can authenticate external without a real password and select inbox) but TB doesn't, then something definitely wrong with your tb setup or with tb itself.

Re:
https://techdrawer.wordpress.com/2014/10/07/testing-ssl-with-your-certificate-using-s_client/
https://serverfault.com/questions/1107751/how-to-setup-dovecot-to-accept-client-certificates-signed-with-a-private-ca-when
https://mcilis.medium.com/how-to-create-a-self-signed-client-certificate-with-openssl-c4af9ac03e99

(In reply to gene smith from comment #11)

Here's how I record the imap log from bash command line:
MOZ_LOG=IMAP:5,timestamp,sync MOZ_LOG_FILE=~/tblog ~/Downloads/thunderbird/thunderbird --allow-downgrade -p

I did not see the "sync" option in the manual I was reading, and I was wary of it but I just assumed that since it's a debug log, maybe they just flush() it by default. I also assumed that the emptiness of the file was because it never got past the SSL negotiation into actually talking IMAP... My bad.

If you restart tb after running with the env vars set, the log get written back to empty and then the new session is recorded. Maybe that's what's happening?

Now that I have the "sync" option, I'm getting some logging:

2022-08-24 09:23:18.172972 UTC - [Parent 18816: Main Thread]: I/IMAP queuing url:imap://Aleksi.Suhonen@imaps.axu.tm:993/select>.Drafts
2022-08-24 09:23:18.173625 UTC - [Parent 18816: IMAP]: D/IMAP ImapThreadMainLoop entering [this=7f79436c8600]
2022-08-24 09:23:18.366779 UTC - [Parent 18816: Main Thread]: I/IMAP queuing url:imap://Aleksi.Suhonen@imaps.axu.tm:993/select>.INBOX
2022-08-24 09:23:18.593610 UTC - [Parent 18816: Main Thread]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-08-24 09:23:18.593757 UTC - [Parent 18816: Main Thread]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:SetupSinkProxy: got m_imapMailFolderSink
2022-08-24 09:23:18.593831 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:ProcessCurrentURL: entering
2022-08-24 09:23:18.593860 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:ProcessCurrentURL:imap://Aleksi%2ESuhonen@imaps.axu.tm:993/discoverallboxes:  = currentUrl
2022-08-24 09:23:19.308568 UTC - [Parent 18816: IMAP]: D/IMAP ReadNextLine [rv=0x804b0014 stream=7f793b41e7e0 nb=0 needmore=1]
2022-08-24 09:23:19.308650 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:CreateNewLineFromSocket: clearing IMAP_CONNECTION_IS_OPEN - rv = 804b0014
2022-08-24 09:23:19.536789 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:TellThreadToDie: close socket connection
2022-08-24 09:23:19.536848 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:CreateNewLineFromSocket: (null)
2022-08-24 09:23:19.536865 UTC - [Parent 18816: IMAP]: D/IMAP SetConnectionStatus(0x804b0014)
2022-08-24 09:23:19.536887 UTC - [Parent 18816: IMAP]: D/IMAP URL failed with code 0x804b0014 (imap://Aleksi%2ESuhonen@imaps.axu.tm:993/discoverallboxes)
2022-08-24 09:23:19.707604 UTC - [Parent 18816: IMAP]: I/IMAP 7f79436c8600:imaps.axu.tm:NA:ProcessCurrentURL: aborting queued urls
2022-08-24 09:23:19.923728 UTC - [Parent 18816: IMAP]: D/IMAP ImapThreadMainLoop leaving [this=7f79436c8600]

From comment 0:
< so the server gives error 42 (did not present cert).

What kind of server is this? Maybe dovecot like I'm testing with?

Yes, it's Dovecot. It's logging this for all the failed cases:

Aug 24 10:32:38 imaps dovecot: imap-login: Disconnected: Connection closed: SSL_accept() failed: error:0A000412:SSL routines::sslv3 alert bad certificate: SSL alert number 42 (client didn't send a cert): user=<>, rip=2001:db8::9, lip=2001:db8::3e1, TLS handshaking: SSL_accept() failed: error:0A000412:SSL routines::sslv3 alert bad certificate: SSL alert number 42

(I censored IP addresses, this log coincides with the thunderbird log below.)

In server setting, you are set to:

Connection Security: SSL/TLS
Authentication Method: TLS Certificate

No, I've had Plaintext auth for years, I don't even remember seeing the TLS Cert auth option before in Thunderbird. I tried changing this, but it's still not working. I think Dovecot expects some password anyway, even if it just ignores the password.
This is what the log looks like after this change:

2022-08-24 10:32:37.732508 UTC - [Parent 19304: Main Thread]: I/IMAP queuing url:imap://Aleksi.Suhonen@imaps.axu.tm:993/select>.INBOX
2022-08-24 10:32:38.010907 UTC - [Parent 19304: Main Thread]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:SetupWithUrlCallback: clearing IMAP_CONNECTION_IS_OPEN
2022-08-24 10:32:38.011037 UTC - [Parent 19304: Main Thread]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:SetupSinkProxy: got m_imapMailFolderSink
2022-08-24 10:32:38.011094 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:ProcessCurrentURL: entering
2022-08-24 10:32:38.011115 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:ProcessCurrentURL:imap://Aleksi%2ESuhonen@imaps.axu.tm:993/discoverallboxes:  = currentUrl
2022-08-24 10:32:38.420361 UTC - [Parent 19304: IMAP]: D/IMAP ReadNextLine [rv=0x804b0014 stream=7fcb12d70700 nb=0 needmore=1]
2022-08-24 10:32:38.420447 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:CreateNewLineFromSocket: clearing IMAP_CONNECTION_IS_OPEN - rv = 804b0014
2022-08-24 10:32:39.045777 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:TellThreadToDie: close socket connection
2022-08-24 10:32:39.045834 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:CreateNewLineFromSocket: (null)
2022-08-24 10:32:39.045851 UTC - [Parent 19304: IMAP]: D/IMAP SetConnectionStatus(0x804b0014)
2022-08-24 10:32:39.045871 UTC - [Parent 19304: IMAP]: D/IMAP URL failed with code 0x804b0014 (imap://Aleksi%2ESuhonen@imaps.axu.tm:993/discoverallboxes)
2022-08-24 10:32:39.484633 UTC - [Parent 19304: IMAP]: I/IMAP 7fcb1b4c3100:imaps.axu.tm:NA:ProcessCurrentURL: aborting queued urls
2022-08-24 10:32:39.844675 UTC - [Parent 19304: IMAP]: D/IMAP ImapThreadMainLoop leaving [this=7fcb1b4c3100]

I wonder if there's some SSL/TLS target I could add into the MOZ_LOG? I couldn't find it in the manual I was reading. I thought "negotiateauth" would have been something like that, but it doesn't seem to be logging anything at all.

For SSL, possibly https://wiki.mozilla.org/NSS:Tracing could give you something.

(In reply to Aleksi Suhonen from comment #0)

I ran into a similar problem with Firefox v102, but I was able to fix it there by fiddling with about:config, but I couldn't find similar settings in TB's about:config. I don't remember which setting it was that fixed it, but I think it was something to do with SSL renegotiation. I think it was "security.tls.enable_post_handshake_auth"...

FWIW, I upgraded my Firefox package in Debian from 102.0.1-3 to 103.0.2-2 today, and cert auth broke for Firefox again. Firefox is reporting some problem with the certificate, but before it can actually tell me what the problem is, the error is replaced with SSL_ERROR_RX_CERTIFICATE_REQUIRED_ALERT, and never asks for a certificate.

I'm guessing the other error is SSL_ERROR_BAD_CERT_DOMAIN, which is confusing since the View Certificate shows the right domain. This guess is based on the fact that it gives the same error for https://www.axu.tm/ which uses the same CA and has a similar setup, but no cert auth requirement.

The "security.tls.enable_post_handshake_auth" setting seems to have reset again, and toggling it doesn't help.

I found Debian/experimental firefox package version 104.0-1 and tried our cert auth intra website with it too. Problem still persists.

I've always just used the "sync" option when I record a log but really never noticed it actually doing anything special like advertised.

No, I've had Plaintext auth for years, I don't even remember seeing the TLS Cert auth option before in Thunderbird. I tried changing this, but it's still not working. I think Dovecot expects some password anyway, even if it just ignores the password.

With TLS Cert authentication, tb just sends your configured username base64 encoded and no password. Dovecot ignores it like this connection to my dovecot:

2022-08-24 16:22:57.204806 UTC - [Parent 9734: IMAP]: I/IMAP 7efe62518800:wally.dbnet.lan:NA:SendData: 52 authenticate EXTERNAL Z2VuZQ==
2022-08-24 16:22:59.182690 UTC - [Parent 9734: IMAP]: I/IMAP 7efe62518800:wally.dbnet.lan:NA:CreateNewLineFromSocket: 52 OK [CAPABILITY IMAP4rev1 ... RIGHTS=texk] Logged in

No password is send. I don't think tb will even try to use the client certificate for authentication unless auth method is set to "TLS Certificate", but I could be wrong.
With "TLS Certificate" authentication, you should see an "authenticate EXTERNAL" like above in your IMAP log.
The log fragment above just shows the connection being dropped, probably due to TLS handshake problems.

https://www.axu.tm/

I'm able to access this in private window after overriding the "unknown issuer" error in firefox 103 from ubuntu 20.04. The handshake pref is at the default.

https://wiki.mozilla.org/NSS:Tracing

This may be a bit out of date. References to NSPR_LOG_MODULES should now be just MOZ_LOG.

Were you able to connect to your dovecot server using the openssl command shown in comment 13 ?

Also, is it possible for you to provide a temporary test account on your dovecot server that I can attempt to access with TB from here? I guess I would need the appropriate client certificate to import and not sure what else.

About comment #13:

I initially skipped that, because everything works with older Thunderbird versions and has been working for a decade. Now I found time to follow those instructions, and SSL handshake worked, but I was not able to adapt the IMAP commands correctly as I don't know that protocol well enough to attempt a manual PLAIN authentication.

However, the error message from the Dovecot logs looks like this:

Sep 12 02:03:49 imaps dovecot: imap-login: Disconnected: Connection closed (auth failed, 2 attempts in 36 secs): user=<Aleksi.Suhonen@axu.tm>, method=PLAIN, rip=2001:db8::c1a9, lip=2001:db8::3e1, TLS: Connection closed

Since the error message has the correct "user=" snatched from my pkcs certificate, I conclude that the SSL authentication part works perfectly with openssl s_client. Compare with the error message in comment #14, where Dovecot complains that no certificate was presented and thus doesn't know the "user=".

Update on comment #16:

I turned off "SSLVerifyClient require" on the server for a second, and I got the webpages I was supposed to get. Firefox (v104 from Debian) made no complaints about certificates presented by the server. But when I turned "SSLVerifyRequire" back on, I still got the same SSL_ERROR_RX_CERTIFICATE_REQUIRED_ALERT.

(In reply to Aleksi Suhonen from comment #20)

Update on comment #16:

I turned off "SSLVerifyClient require" on the server for a second, and I got the webpages I was supposed to get. Firefox (v104 from Debian) made no complaints about certificates presented by the server. But when I turned "SSLVerifyRequire" back on, I still got the same SSL_ERROR_RX_CERTIFICATE_REQUIRED_ALERT.

Well ... I figured this one out, and it was possibly a bit embarrassing: I had deleted my valid user certificate from the browser cert store, when I had meant to delete an expired user certificate. After I loaded the valid user cert into Firefox again, it started working.

I went back to double check my Thunderbird certificates, but I've got the correct and valid certificates there, so original issue still remains.

Whiteboard: tb-crypto-broken-feature

I recreated the IMAP server certificate with the modern subject-alternative-name extension, and I can now use thunderbird 102 and newer with it. I suppose my issue has now been resolved.

I would like to say tho that the client should give more feedback on what is going wrong, before I feel it's ready for release.

So perhaps bug 1777717?

(In reply to Magnus Melin [:mkmelin] from comment #23)

So perhaps bug 1777717?

Looks like that, except TB wasn't complaining at all, it was just failing to connect with the loading animation continuing indefinitely.

I guess it's a bit different symptom for TLS auth yeah.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.