Closed Bug 1570222 Opened 4 months ago Closed 2 months ago

dotnet core 2.2 dev ssl cert no longer work with error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

Categories

(Core :: Security: PSM, defect, P1)

69 Branch
Desktop
Windows 10
defect

Tracking

()

RESOLVED FIXED
mozilla71
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 70+ fixed
firefox67 --- unaffected
firefox68 --- unaffected
firefox69 --- wontfix
firefox70 + fixed
firefox71 + verified

People

(Reporter: p.hermann, Assigned: keeler, NeedInfo)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [psm-assigned])

Attachments

(9 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

i created a new dotnet core ssl dev cert with
dotnet dev-certs https --trust
A new SSL Certificate was created and trusted
After starting the dotnet core app i opened firefox dev 69.0b4 (64-Bit)
And navigate to https://localhost:5001/swagger/index.html

Actual results:

i get the error:

Fehler: Gesicherte Verbindung fehlgeschlagen

Beim Verbinden mit localhost:5001 trat ein Fehler auf. Das Zertifikat enthält eine unbekannte kritische Erweiterung. Fehlercode: SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

Die Website kann nicht angezeigt werden, da die Authentizität der erhaltenen Daten nicht verifiziert werden konnte.
Kontaktieren Sie bitte den Inhaber der Website, um ihn über dieses Problem zu informieren.

Weitere Informationen…

Expected results:

The websites opens and shows the swagger docu page.
On Firefox 60.6.3esr (32-bit) it is working.

My System:
Windows 10 64 Bit Version 10.0.15063
Dotnet Core SDK 2.2.108
Visual Studio 2017 Pro 15.9.14
Visual Studio Code v 1.36.1

Hi Hermann,

Thanks for the details, for obvious reasons I'm not being able to reproduce this bug but I've chosen a component. If you consider there's a more proper one, feel free to change it. We'll await the devs' response.

Best regards, Flor.

Component: Untriaged → General
Product: Firefox → WebExtensions
Version: 69 Branch → Firefox 69
Component: General → Security: PSM
Product: WebExtensions → Core
Version: Firefox 69 → 69 Branch

Can you attach an example certificate that doesn't work to this bug? Thanks!

Flags: needinfo?(p.hermann)
Attached file devcert-core.cer

The Cert that does not work

Flags: needinfo?(p.hermann)

For that certificate to work in Firefox, you'll have to add it as a trust anchor. Since it's a self-signed end-entity certificate, this is only possible through the command-line using the NSS utility certutil (note that this is distinct from the Windows utility certutil). You can trust it like so: certutil -A -d [path to your Firefox profile] -n [nickname] -t "CT,," -i [path to the certificate file]. Another option would be to create a CA certificate, import it into Firefox using the certificate manager (about:preferences -> search for "Certificates", click "View Certificates", click "Authorities", click "Import"), and use it to sign your server certificate.

Status: UNCONFIRMED → RESOLVED
Closed: 3 months ago
Resolution: --- → INVALID

Where can i get a compiled version from nss certutil?

Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---

https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&searchStr=build-win64-shippable%2Fopt should show you the windows 64-bit builds (click the first green +3). If you select the build job (B) and click Job Details, there should be a link to a file called target.common.tests.tar.gz. If you extract that, certutil.exe is in the bin directory.

Status: UNCONFIRMED → RESOLVED
Closed: 3 months ago3 months ago
Resolution: --- → INVALID

This is still not working. See Picture for details.

Status: RESOLVED → UNCONFIRMED
OS: Unspecified → Windows 10
Hardware: Unspecified → Desktop
Resolution: INVALID → ---

I have the same problem with several self-signed certificates that worked fine with FF 68. Since the update to 69.0 I get the error (Windows 10).

(In reply to p.hermann from comment #8)

Created attachment 9090595 [details]
I cant do this with certutil

This is still not working. See Picture for details.

Try downloading target.zip as well - the libraries are in the firefox/ directory.

(In reply to hapeka from comment #9)

I have the same problem with several self-signed certificates that worked fine with FF 68. Since the update to 69.0 I get the error (Windows 10).

Can you attach an example certificate that doesn't work?

Flags: needinfo?(hapeka)

I have attached the server.crt as an example for certificates that don't work any longer

Flags: needinfo?(hapeka)

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

(In reply to hapeka from comment #11)

Created attachment 9090937 [details]
server.crt - self-signed certificate which provoces the error msg in FF 69

I have attached the server.crt as an example for certificates that don't work any longer

Are you sure that's the server certificate? That looks more like a CA certificate (note it has an extension basic constraints: CA: true and no subject alternative names extension).

Flags: needinfo?(hapeka)

(In reply to Sander Goudswaard from comment #12)

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

The error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION is not overridable in Firefox, so something else is going on. Can you attach the server certificate and the CA certificate that issued the server certificate? It might also help to attach a packet trace of the TLS handshake.

Flags: needinfo?(sander+bugzilla)

We have the same problem with many self signed certificates, e. g. our Pfsense firewall and many other internal systems.

I attached an example certificate too, hope it helps.

Selfsigned cert attached.

Flags: needinfo?(sander+bugzilla)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #14)

The error SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION is not overridable in Firefox, so something else is going on. Can you attach the server certificate and the CA certificate that issued the server certificate? It might also help to attach a packet trace of the TLS handshake.

I created the certificate with a command similar to this:
openssl req -x509 -sha256 -nodes -days 730 -newkey rsa:2048 -keyout server.key -out server.crt
I didn't check exactly which extensions it has, just wanted something I could use on my test system.

Flags: needinfo?(hapeka)

(In reply to Sander Goudswaard from comment #12)

Found this bug report because many of our internal certificates stopped working in Firefox 69, giving the SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION error message. They still work in Chrome (after the certificate warning, you can continue in Chrome but not in Firefox.

Workaround for me is to go to about:preferences#privacy and view the certificates, then on the Servers tab enter the URL to the server. The certificate is then fetched and can be stored, after which Firefox can connect again.
This should have the same result as the checkbox on certificate warning pages in Firefox 68, that would store the exception.

Exactly the same issue for me in a corporate environment.
Just workaround doesn't work for me.
More information about my case on the Firefox support forums:
https://support.mozilla.org/en-US/questions/1264964#answer-1253012

Sample certificate attached.

Attached file Synology.cer

If you create a new Firefox profile, do you see the same error?

Flags: needinfo?(p.hermann)

Also, are you using the enterprise roots feature? (security.enterprise_roots.enabled set to true or via enterprise policy)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #22)

If you create a new Firefox profile, do you see the same error?

Yes

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #23)

Also, are you using the enterprise roots feature? (security.enterprise_roots.enabled set to true or via enterprise policy)

Yes

Do you see the same error if you disable enterprise roots?
If not, can you do the following with enterprise roots enabled:

  1. set devtools.chrome.enabled to true in about:config
  2. open the browser console (ctrl + shift + j)
  3. run Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseRoots() in the browser console
  4. right-click on the output from that, select copy object, paste the contents into a file, and either email that to me or attach it to this bug (it will contain the bytes of the 3rd party root certificates that Firefox found in your Windows registry)
  5. do the same with Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseIntermediates()

(please see comment 25)

Flags: needinfo?(sander+bugzilla)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #25)

Do you see the same error if you disable enterprise roots?

No. So I proceeded with your instructions and sent the files via e-mail to you.

Flags: needinfo?(sander+bugzilla)

Network guys over here were encountering same problem since MitM was rolled out. They were unable to access the local Nutanix management console which used a self-signed cert.
STR:
① Create new firefox profile.
② Load local network Nutanix console.
③ Click through to accept self-signed cert. Console loads
④ Open a second tab in this clean Firefox profile, access Google.com. Firefox errors briefly due to the corporate MitM then autoenables the corporate spying override.
⑤ Go back to first tab, hit reload. SEC_ERROR_UNKNOWN_CRITICAL_EXTENSION

One possible wrinkle that could impact your fixes is that the local servers are not running on a private IPv4 range. They are however running on the same range as the user's Firefox.

So. At present the only fixes are a dedicated Firefox profile that only accesses these servers and nothing else, or manually adding all the server certs to the Firefox store?

Priority: -- → P2
Whiteboard: [psm-backlog]

I've been encountering this a lot now with our various dev servers where self-signed certs are not uncommon. Besides a separate profile, leaving about:config open in a separate tab and setting security.enterprise_roots.enabled;false whenever I need to access the self-signed server works too. I just have to reenable it when I need to go to other websites.

Anyone who can reproduce this: finding when this stopped working would be helpful. Can you use https://mozilla.github.io/mozregression/quickstart.html to narrow this down? Thanks!

Flags: needinfo?(bugs)

I have used the mozregression, but the outcome confuses me. I couldn't reproduce the problem in any version I tried, even very recent ones.
Then I tried FF nightly, but still couldn't reproduce it.
But if I just create a new profile in my standard FF installation the error is reproducible.

Please ignore my previous comment. I was confused by the fact that this error depends on the enterprise roots feature being enabled. It is enabled in my standard FF, but it wasn't in the other versions.
I did the bisection again, with the enterprise roots feature turned on. And this finally led me to this commit:
https://hg.mozilla.org/integration/autoland/rev/5a721a7648f2db40785729ed8fc7c7444c1afcaf
(Bug 1551177 - avoid searching unproductive certificate paths during verification r=jcj,KevinJacobs)

Regressed by: 1551177

Awesome! Can you try one of these builds to see if I've correctly understood the issue?
https://treeherder.mozilla.org/#/jobs?repo=try&revision=894992f343de76333492f8218260632d8fdd127d
(click on the appropriate green B, click Job Details, download target.installer.exe)
Thanks!

Flags: needinfo?(hapeka)

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #33)

Awesome! Can you try one of these builds to see if I've correctly understood the issue?

I clicked 'Bs' behind 'Windows 2012 x64 opt' and downloaded target.zip, verified that enterprise roots are still enabled (it is because the setting is distributed via group policy). The issue does not reproduce in that build.

Thanks!
Does this bug reproduce in ESR 68?

Flags: needinfo?(sander+bugzilla)
Priority: P2 → P1
Whiteboard: [psm-backlog] → [psm-assigned]

During path building, mozilla::pkix filters out candidate certificates provided
by trust domains where the subject distinguished name does not match the issuer
distinguished name of the certificate it's trying to find an issuer for.
However, if there's a problem decoding the candidate issuer certificate,
mozilla::pkix will make a note of this error, regardless of if that certificate
was potentially a suitable issuer. If no trusted path is found, the error from
that unrelated certificate may ultimately be returned by mozilla::pkix,
resulting in confusion.

Before this patch, NSSCertDBTrustDomain could cause this behavior by blithely
passing every known 3rd party certificate to mozilla::pkix (other sources of
certificates already filter on subject distinguished name). This patch adds
filtering to 3rd party certificates as well.

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #35)

Thanks!
Does this bug reproduce in ESR 68?

Tested in ESR 68.0.1, does not reproduce.

Flags: needinfo?(sander+bugzilla)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2a56f91a20b2
avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r=kjacobs
Status: UNCONFIRMED → RESOLVED
Closed: 3 months ago2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Assignee: nobody → dkeeler

Is this something we should consider for Beta uplift or can this ride Fx71 to release?

Yes. ESR as well, if possible.

Flags: needinfo?(dkeeler)

Per comment 37, isn't ESR unaffected?

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

Beta/Release Uplift Approval Request

  • User impact if declined: Under pessimal configurations, users will not be able to override certificate errors for administrative interfaces on devices like routers, etc.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Small, localized patch. Logically, we're doing an operation that will always happen anyway, so it shouldn't introduce any new errors.
  • String changes made/needed:

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Enterprise-facing feature
  • User impact if declined: Under pessimal configurations, users will not be able to override certificate errors for administrative interfaces on devices like routers, etc.
  • Fix Landed on Version:
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Small, localized patch. Logically, we're doing an operation that will always happen anyway, so it shouldn't introduce any new errors.
  • String or UUID changes made by this patch:
Attachment #9098665 - Flags: approval-mozilla-esr68?
Attachment #9098665 - Flags: approval-mozilla-beta?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #42)

Per comment 37, isn't ESR unaffected?

If my understanding is correct, it is affected. Bug 1473573 essentially introduced this (on Windows - MacOS has probably been affected for even longer). Bug 1551177 made it easier to get into a state where this will happen by limiting the number of potential issuers the cert verifier passes to mozilla::pkix.

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

In that case, ESR68 will need a rebased patch :)

Flags: needinfo?(dkeeler)
Attachment #9098665 - Flags: approval-mozilla-esr68?

Comment on attachment 9098665 [details]
bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r?kjacobs

Fix for a regression from 69, let's uplift for beta 14.

Attachment #9098665 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attached patch patch for esr68Splinter Review

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: (see comment 43)
  • User impact if declined:
  • Fix Landed on Version:
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String or UUID changes made by this patch:
Flags: needinfo?(dkeeler)
Attachment #9100040 - Flags: review+
Attachment #9100040 - Flags: approval-mozilla-esr68?
Comment on attachment 9100040 [details] [diff] [review]
patch for esr68

Approved for 68.2esr.
Attachment #9100040 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Flags: qe-verify+
QA Whiteboard: [qa-triaged]

Hello P.Hermann, since we do not have the environment set up for this issue can you please take a look in our latest Beta and Nightly builds and recheck if this issue is Fixed on your end? you can find the builds here:

Nightly - https://nightly.mozilla.org/
BETA - https://www.mozilla.org/ro/firefox/channel/desktop/

On Firefox Nightly 71.0a1 it is now working.
On Firefox Developer 70.0b14 it is still not working.

Flags: needinfo?(p.hermann)

Hi P.Hermann, thanks for taking the time to verify this issue, @dkeeler can you please take a look at this for Beta 70.0b14 ? it seems the issue still occurs there.

I will update the flags for Nightly 71.0a1.

Flags: needinfo?(dkeeler)

Is the behavior any different with a new profile?

Flags: needinfo?(dkeeler) → needinfo?(p.hermann)

it was a complete fresh installation on a new system.

Flags: needinfo?(p.hermann)

Can you please attach to this bug or send me a packet trace of the TLS handshake that fails, the cert9.db file in the Firefox profile directory, and the output of:

  • set devtools.chrome.enabled to true in about:config
  • open the browser console (ctrl + shift + j)
  • run Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseRoots() in the browser console
  • right-click on the output from that, select copy object, paste the contents into a file, and either email that to me or attach it to this bug (it will contain the bytes of the 3rd party root certificates that Firefox found in your Windows registry)
  • do the same with Cc["@mozilla.org/psm;1"].getService(Ci.nsINSSComponent).getEnterpriseIntermediates()

Thanks!

Flags: needinfo?(p.hermann)

Sorry i cant.
After my windows was reseted another time, i have installed Firefox71.0b1 and there it is working.

Flags: needinfo?(p.hermann)

Just FYI, I retested in latest nightly in a clean profile on our corporate network.

  • self-signed certs work, with or without enterprise roots enabled
  • accessing many internet websites autoenables enterprise roots as the pref requires (such as my own Let's Encrypt signed domain).
  • https://google.com is BROKEN with or without enterprise roots enabled. and the autoenable is as well. Related to google's pinning? Anyway this is NEW in nightly. Release is fine. Errors with SSL_RX_MALFORMED_SERVER_HELLO.

I'm guessing it's a regression from this bug. Shame, was excitedly trying to get my coworkers back on Firefox and my demo fell rather flat.

Flags: needinfo?(bugs)

p.s. - sorry, was trying to get this test feedback in earlier before you guys marked the bug FIXED, but BMO disabled my account due to a temporary mail error and I thought that asking on #bmo would be more reliable than firing an email into the void, which it wasn't. #bmo used to work a lot better...

hm... p.p.s.
Just tested nightly from 2019-09-01 and it is also broken with https://google.com with the same error as above - it also failed to autoenable the enterprise roots, but manually enabling fixed other websites (tested on https://m8y.org which the 2019-10-22 nightly autoenables correctly but the 2019-09-01 nightly failed to do so on a clean profile).

So. I have no idea what's going on now, and whether this problem is part of this bug report's broken enterprise certs triggered by bug #1473573 or something totally different. But release is fine.

I figured it was worth bringing up in the closed bug though, since you guys were backporting stuff to stable and ESR and I don't want to lose all functioning copies of Firefox.

Nemo - please file a new bug with as much information you can include: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM

Flags: needinfo?(bugs)
Duplicate of this bug: 1585821

Alright. Will do. It'll be basically the comments above but with hopefully clearer STR.
Basically I was just panicked 'cause this bug was approved for backport and I didn't want to lose access to google etc on all my firefox installs (although I guess in a pinch I could just refuse the update to ESR/stable).

Flags: needinfo?(bugs)
You need to log in before you can comment on or make changes to this bug.