1570222 - dotnet core 2.2 dev ssl cert no longer work with error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[p.hermann]

Attachment: 2019-07-31 08_49_46-Seiten-Ladefehler - Firefox Developer Edition.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

i created a new dotnet core ssl dev cert with
dotnet dev-certs https --trust
A new SSL Certificate was created and trusted
After starting the dotnet core app i opened firefox dev 69.0b4 (64-Bit)
And navigate to https://localhost:5001/swagger/index.html

Actual results:

i get the error:

Fehler: Gesicherte Verbindung fehlgeschlagen

Beim Verbinden mit localhost:5001 trat ein Fehler auf. Das Zertifikat enthält eine unbekannte kritische Erweiterung. Fehlercode: SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

Die Website kann nicht angezeigt werden, da die Authentizität der erhaltenen Daten nicht verifiziert werden konnte.
Kontaktieren Sie bitte den Inhaber der Website, um ihn über dieses Problem zu informieren.

Weitere Informationen…

Expected results:

The websites opens and shows the swagger docu page.
On Firefox 60.6.3esr (32-bit) it is working.

Comment 1

[p.hermann]

My System:
Windows 10 64 Bit Version 10.0.15063
Dotnet Core SDK 2.2.108
Visual Studio 2017 Pro 15.9.14
Visual Studio Code v 1.36.1

Comment 2

[Florencia Di Ciocco, NI to vbalducci]

Hi Hermann,

Thanks for the details, for obvious reasons I'm not being able to reproduce this bug but I've chosen a component. If you consider there's a more proper one, feel free to change it. We'll await the devs' response.

Best regards, Flor.

Comment 3

[Dana Keeler (she/her) [:keeler]]

Can you attach an example certificate that doesn't work to this bug? Thanks!

Comment 4

[p.hermann]

Attachment: devcert-core.cer

The Cert that does not work

Comment 5

[Dana Keeler (she/her) [:keeler]]

For that certificate to work in Firefox, you'll have to add it as a trust anchor. Since it's a self-signed end-entity certificate, this is only possible through the command-line using the NSS utility certutil (note that this is distinct from the Windows utility certutil). You can trust it like so: certutil -A -d [path to your Firefox profile] -n [nickname] -t "CT,," -i [path to the certificate file]. Another option would be to create a CA certificate, import it into Firefox using the certificate manager (about:preferences -> search for "Certificates", click "View Certificates", click "Authorities", click "Import"), and use it to sign your server certificate.

Comment 6

[p.hermann]

Where can i get a compiled version from nss certutil?

Comment 7

[Dana Keeler (she/her) [:keeler]]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&searchStr=build-win64-shippable%2Fopt should show you the windows 64-bit builds (click the first green +3). If you select the build job (B) and click Job Details, there should be a link to a file called target.common.tests.tar.gz. If you extract that, certutil.exe is in the bin directory.

Comment 8

[p.hermann]

Attachment: I cant do this with certutil

This is still not working. See Picture for details.

Comment 9

[hapeka]

I have the same problem with several self-signed certificates that worked fine with FF 68. Since the update to 69.0 I get the error (Windows 10).

Comment 10

[Dana Keeler (she/her) [:keeler]]

(In reply to p.hermann from comment #8)

Created attachment 9090595 [details]
I cant do this with certutil

This is still not working. See Picture for details.

Try downloading target.zip as well - the libraries are in the firefox/ directory.

(In reply to hapeka from comment #9)

I have the same problem with several self-signed certificates that worked fine with FF 68. Since the update to 69.0 I get the error (Windows 10).

Can you attach an example certificate that doesn't work?

Comment 11

[hapeka]

Attachment: server.crt - self-signed certificate which provoces the error msg in FF 69

I have attached the server.crt as an example for certificates that don't work any longer

Comment 12

[Sander Goudswaard]

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

Comment 13

[Dana Keeler (she/her) [:keeler]]

(In reply to hapeka from comment #11)

Created attachment 9090937 [details]
server.crt - self-signed certificate which provoces the error msg in FF 69

I have attached the server.crt as an example for certificates that don't work any longer

Are you sure that's the server certificate? That looks more like a CA certificate (note it has an extension basic constraints: CA: true and no subject alternative names extension).

Comment 14

[Dana Keeler (she/her) [:keeler]]

(In reply to Sander Goudswaard from comment #12)

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

The error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION is not overridable in Firefox, so something else is going on. Can you attach the server certificate and the CA certificate that issued the server certificate? It might also help to attach a packet trace of the TLS handshake.

Comment 15

[Sebastian Koch]

We have the same problem with many self signed certificates, e. g. our Pfsense firewall and many other internal systems.

I attached an example certificate too, hope it helps.

Comment 16

[Sebastian Koch]

Attachment: webConfigurator+default+(5bd8c1bedc85f).crt

Comment 17

[Sander Goudswaard]

Attachment: reply_comment14_selfsigned.cer

Selfsigned cert attached.

Comment 18

[Sander Goudswaard]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #14)

The error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION is not overridable in Firefox, so something else is going on. Can you attach the server certificate and the CA certificate that issued the server certificate? It might also help to attach a packet trace of the TLS handshake.

Comment 19

[hapeka]

I created the certificate with a command similar to this:
openssl req -x509 -sha256 -nodes -days 730 -newkey rsa:2048 -keyout server.key -out server.crt
I didn't check exactly which extensions it has, just wanted something I could use on my test system.

Comment 20

[prassasandreas]

(In reply to Sander Goudswaard from comment #12)

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

Exactly the same issue for me in a corporate environment.
Just workaround doesn't work for me.
More information about my case on the Firefox support forums:
https://support.mozilla.org/en-US/questions/1264964#answer-1253012

Sample certificate attached.

Comment 21

[prassasandreas]

Attachment: Synology.cer

Comment 22

[Dana Keeler (she/her) [:keeler]]

If you create a new Firefox profile, do you see the same error?

Comment 23

[Dana Keeler (she/her) [:keeler]]

Also, are you using the enterprise roots feature? (security.enterprise_roots.enabled set to true or via enterprise policy)

Comment 24

[Sander Goudswaard]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #22)

If you create a new Firefox profile, do you see the same error?

Yes

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #23)

Also, are you using the enterprise roots feature? (security.enterprise_roots.enabled set to true or via enterprise policy)

Yes

Comment 25

[Dana Keeler (she/her) [:keeler]]

Do you see the same error if you disable enterprise roots?
If not, can you do the following with enterprise roots enabled:

1. set devtools.chrome.enabled to true in about:config
2. open the browser console (ctrl + shift + j)
3. run Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseRoots() in the browser console
4. right-click on the output from that, select copy object, paste the contents into a file, and either email that to me or attach it to this bug (it will contain the bytes of the 3rd party root certificates that Firefox found in your Windows registry)
5. do the same with Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseIntermediates()

Comment 26

[Dana Keeler (she/her) [:keeler]]

(please see comment 25)

Comment 27

[Sander Goudswaard]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #25)

Do you see the same error if you disable enterprise roots?

No. So I proceeded with your instructions and sent the files via e-mail to you.

Comment 28

[nemo]

Network guys over here were encountering same problem since MitM was rolled out. They were unable to access the local Nutanix management console which used a self-signed cert.
STR:
① Create new firefox profile.
② Load local network Nutanix console.
③ Click through to accept self-signed cert. Console loads
④ Open a second tab in this clean Firefox profile, access Google.com. Firefox errors briefly due to the corporate MitM then autoenables the corporate spying override.
⑤ Go back to first tab, hit reload. SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

One possible wrinkle that could impact your fixes is that the local servers are not running on a private IPv4 range. They are however running on the same range as the user's Firefox.

So. At present the only fixes are a dedicated Firefox profile that only accesses these servers and nothing else, or manually adding all the server certs to the Firefox store?

Comment 29

[nemo]

I've been encountering this a lot now with our various dev servers where self-signed certs are not uncommon. Besides a separate profile, leaving about:config open in a separate tab and setting security.enterprise_roots.enabled;false whenever I need to access the self-signed server works too. I just have to reenable it when I need to go to other websites.

Comment 30

[Dana Keeler (she/her) [:keeler]]

Anyone who can reproduce this: finding when this stopped working would be helpful. Can you use https://mozilla.github.io/mozregression/quickstart.html to narrow this down? Thanks!

Comment 31

[hapeka]

I have used the mozregression, but the outcome confuses me. I couldn't reproduce the problem in any version I tried, even very recent ones.
Then I tried FF nightly, but still couldn't reproduce it.
But if I just create a new profile in my standard FF installation the error is reproducible.

Comment 32

[hapeka]

Please ignore my previous comment. I was confused by the fact that this error depends on the enterprise roots feature being enabled. It is enabled in my standard FF, but it wasn't in the other versions.
I did the bisection again, with the enterprise roots feature turned on. And this finally led me to this commit:
https://hg.mozilla.org/integration/autoland/rev/5a721a7648f2db40785729ed8fc7c7444c1afcaf
(Bug 1551177 - avoid searching unproductive certificate paths during verification r=jcj,KevinJacobs)

Comment 33

[Dana Keeler (she/her) [:keeler]]

Awesome! Can you try one of these builds to see if I've correctly understood the issue?
https://treeherder.mozilla.org/#/jobs?repo=try&revision=894992f343de76333492f8218260632d8fdd127d
(click on the appropriate green B, click Job Details, download target.installer.exe)
Thanks!

Comment 34

[Sander Goudswaard]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #33)

Awesome! Can you try one of these builds to see if I've correctly understood the issue?

I clicked 'Bs' behind 'Windows 2012 x64 opt' and downloaded target.zip, verified that enterprise roots are still enabled (it is because the setting is distributed via group policy). The issue does not reproduce in that build.

Comment 35

[Dana Keeler (she/her) [:keeler]]

Thanks!
Does this bug reproduce in ESR 68?

Comment 36

[Dana Keeler (she/her) [:keeler]]

Attachment: bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

During path building, mozilla::pkix filters out candidate certificates provided
by trust domains where the subject distinguished name does not match the issuer
distinguished name of the certificate it's trying to find an issuer for.
However, if there's a problem decoding the candidate issuer certificate,
mozilla::pkix will make a note of this error, regardless of if that certificate
was potentially a suitable issuer. If no trusted path is found, the error from
that unrelated certificate may ultimately be returned by mozilla::pkix,
resulting in confusion.

Before this patch, NSSCertDBTrustDomain could cause this behavior by blithely
passing every known 3rd party certificate to mozilla::pkix (other sources of
certificates already filter on subject distinguished name). This patch adds
filtering to 3rd party certificates as well.

Comment 37

[Sander Goudswaard]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #35)

Thanks!
Does this bug reproduce in ESR 68?

Tested in ESR 68.0.1, does not reproduce.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a56f91a20b2
avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r=kjacobs

Comment 39

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/2a56f91a20b2

Comment 40

[Ryan VanderMeulen [:RyanVM]]

Is this something we should consider for Beta uplift or can this ride Fx71 to release?

Comment 41

[Dana Keeler (she/her) [:keeler]]

Yes. ESR as well, if possible.

Comment 42

[Ryan VanderMeulen [:RyanVM]]

Per comment 37, isn't ESR unaffected?

Comment 43

[Dana Keeler (she/her) [:keeler]]

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

Beta/Release Uplift Approval Request

• User impact if declined: Under pessimal configurations, users will not be able to override certificate errors for administrative interfaces on devices like routers, etc.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small, localized patch. Logically, we're doing an operation that will always happen anyway, so it shouldn't introduce any new errors.
• String changes made/needed:

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enterprise-facing feature
• User impact if declined: Under pessimal configurations, users will not be able to override certificate errors for administrative interfaces on devices like routers, etc.
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small, localized patch. Logically, we're doing an operation that will always happen anyway, so it shouldn't introduce any new errors.
• String or UUID changes made by this patch:

Comment 44

[Dana Keeler (she/her) [:keeler]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #42)

Per comment 37, isn't ESR unaffected?

If my understanding is correct, it is affected. Bug 1473573 essentially introduced this (on Windows - MacOS has probably been affected for even longer). Bug 1551177 made it easier to get into a state where this will happen by limiting the number of potential issuers the cert verifier passes to mozilla::pkix.

Comment 45

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

In that case, ESR68 will need a rebased patch :)

Comment 46

[Liz Henry (:lizzard)]

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

Fix for a regression from 69, let's uplift for beta 14.

Comment 47

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/d69eab8be1bf

Comment 48

[Dana Keeler (she/her) [:keeler]]

Attachment: patch for esr68

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: (see comment 43)
• User impact if declined:
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String or UUID changes made by this patch:

Comment 49

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9100040 [details] [diff] [review]
patch for esr68

Approved for 68.2esr.

Comment 50

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/c62c32d9da65

Comment 51

[Rares Doghi, Desktop QA]

Hello P.Hermann, since we do not have the environment set up for this issue can you please take a look in our latest Beta and Nightly builds and recheck if this issue is Fixed on your end? you can find the builds here:

Nightly - https://nightly.mozilla.org/
BETA - https://www.mozilla.org/ro/firefox/channel/desktop/

Comment 52

[p.hermann]

On Firefox Nightly 71.0a1 it is now working.
On Firefox Developer 70.0b14 it is still not working.

Comment 53

[Rares Doghi, Desktop QA]

Hi P.Hermann, thanks for taking the time to verify this issue, @dkeeler can you please take a look at this for Beta 70.0b14 ? it seems the issue still occurs there.

I will update the flags for Nightly 71.0a1.

Comment 54

[Dana Keeler (she/her) [:keeler]]

Is the behavior any different with a new profile?

Comment 55

[p.hermann]

it was a complete fresh installation on a new system.

Comment 56

[Dana Keeler (she/her) [:keeler]]

Can you please attach to this bug or send me a packet trace of the TLS handshake that fails, the cert9.db file in the Firefox profile directory, and the output of:

• set devtools.chrome.enabled to true in about:config
• open the browser console (ctrl + shift + j)
• run Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseRoots() in the browser console
• right-click on the output from that, select copy object, paste the contents into a file, and either email that to me or attach it to this bug (it will contain the bytes of the 3rd party root certificates that Firefox found in your Windows registry)
• do the same with Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseIntermediates()

Thanks!

Comment 57

[p.hermann]

Sorry i cant.
After my windows was reseted another time, i have installed Firefox71.0b1 and there it is working.

Comment 58

[nemo]

Just FYI, I retested in latest nightly in a clean profile on our corporate network.

• self-signed certs work, with or without enterprise roots enabled
• accessing many internet websites autoenables enterprise roots as the pref requires (such as my own Let's Encrypt signed domain).
• https://google.com is BROKEN with or without enterprise roots enabled. and the autoenable is as well. Related to google's pinning? Anyway this is NEW in nightly. Release is fine. Errors with SSL_RX_MALFORMED_SERVER_HELLO.

I'm guessing it's a regression from this bug. Shame, was excitedly trying to get my coworkers back on Firefox and my demo fell rather flat.

Comment 59

[nemo]

p.s. - sorry, was trying to get this test feedback in earlier before you guys marked the bug FIXED, but BMO disabled my account due to a temporary mail error and I thought that asking on #bmo would be more reliable than firing an email into the void, which it wasn't. #bmo used to work a lot better...

Comment 60

[nemo]

hm... p.p.s.
Just tested nightly from 2019-09-01 and it is also broken with https://google.com with the same error as above - it also failed to autoenable the enterprise roots, but manually enabling fixed other websites (tested on https://m8y.org which the 2019-10-22 nightly autoenables correctly but the 2019-09-01 nightly failed to do so on a clean profile).

So. I have no idea what's going on now, and whether this problem is part of this bug report's broken enterprise certs triggered by bug #1473573 or something totally different. But release is fine.

I figured it was worth bringing up in the closed bug though, since you guys were backporting stuff to stable and ESR and I don't want to lose all functioning copies of Firefox.

Comment 61

[Dana Keeler (she/her) [:keeler]]

Nemo - please file a new bug with as much information you can include: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM

Comment 63

[nemo]

Alright. Will do. It'll be basically the comments above but with hopefully clearer STR.
Basically I was just panicked 'cause this bug was approved for backport and I didn't want to lose access to google etc on all my firefox installs (although I guess in a pinch I could just refuse the update to ESR/stable).