Closed Bug 1682881 Opened 4 years ago Closed 4 years ago

Firefox 84 freeze if a pkcs11 module is loaded

Categories

(Core :: Security: PSM, defect)

Product:
Core
Component:
Security: PSM
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Tracking Flags:
Tracking Status
relnote-firefox --- 84+
firefox-esr78 --- unaffected
firefox83 --- unaffected
firefox84 + verified
firefox85 --- fixed
firefox86 --- fixed

People

(Reporter: bugzilla, Assigned: kjacobs)

References

(Regression)

Details

(Keywords: regression)

Crash Data

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

After updating to FF 84, the browser fails to load any HTTPS:// web site.
This happens when a PKCS#11 module is loaded in the Security Devices settings AND the module is an interface to a smart card.

I think that the problem belongs to NSS layer since it's the low level interface to the PKCS#11 modules.

To check it you need to have:

  • a smart card or a similar security token
  • a PKCS#11 module to use it
  1. in Security & privacy settings add a single PKCS#11 module to FF Security Devices
  2. close FF
  3. reopen FF and browse to any HTTPS:// web site
  4. FF never finish load the web site or shows a timeout error after a while
  5. if you go to Security Devices Manager or to "Show certificates", FF completely freeze

Actual results:

Until FF 83 the browser was correctly working with any PKCS#11 module from any vendor.
Today with FF 84 update it stopped working an my machine and the machine of many many other peoples using PKCS#11 modules from many vendor (including but not limited to NXP ID-Protect that I tested personally).

Thanks for the report. This is likely bug 1679290, which was previously not known impact Firefox.

Before I make a point release of NSS for this, could you please confirm that this works as expected with Firefox Beta (85)?

Flags: needinfo?(midori3)

I'm testing FF 85 beta b2 on various machines, it looks like that it's working much better.
But I'm not really sure the problem is completely gone, I had a couple of freeze on 1 machine and I need some time to collect more details.

Flags: needinfo?(midori3)

[Tracking Requested - why for this release]: Regression blocks loading of websites with certain third-party PKCS11 modules and smartcards.

I plan to make a point release (NSS 3.59.1) for this, but I'll give a little time for the reporter to confirm that the fix from bug 1679290 fully resolves the issue.

tracking-firefox84: --- → ?

I wasn't able to reproduce this on Windows with Fx 84.0 and a SafeNet eToken 5110 (both HTTPS and client authentication work via eTPKCS11.dll). If I attempt to use OpenSC instead, I get the behavior described on all versions tested back to Fx70, so that doesn't help...

@J.K.Umeboshi, please let us know if you continue to see problems in 85 Beta that are not present in 83.

Per conversation with :RyanVM, I'll hold on making the NSS point release for now.

Flags: needinfo?(midori3)
Crash Signature: [@ shutdownhang | mozilla::TaskController::GetRunnableForMTTask | mozilla::net::nsHttpConnectionMgr::Shutdown]

I did some other tests and I was able to reproduce the freeze problem also on FF 85.

The OS is Windows 10 64bit.

It looks like there is a race condition triggered by the speed of the token/smart card.

  • SoftHSM2 always works (works fine also on FF 84)
  • OpenSC used with a very simple and really fast eID smart card: Always Works
  • ID Protect (NXP/Athena) smart card: apparently works
  • Bit4id MW with a smart card: apparently works
  • Bit4id MW with a smart card and a very old/slow version: Always freeze

Some smart card device only apparently works.
I think that it depends on the responsiveness of the of the device:

  • If I start FF with the smart card already in the reader, it works (due to MW caching or at least the card already powered on).
  • If I remove and reinsert the smart card, FF freeze after few seconds.

This doesn't happens with OpenSC uset with the eID since it's able to read it really really fast.
I don't have a Safenet Token right now, but I tested it in the past and I can say that it repond almost istantaneously, so I belive that it works.

NOTE:
this bug looks very similar to me to the bug #1101547, that still there.

Flags: needinfo?(midori3)

Thanks. I'm inclined to believe that the Fx85 issues are unrelated:

  1. We've now both reproduced similar problems on other versions of Firefox.
  2. From the data on crash-stats, a high volume of the Fx84 reports show NSS waiting as in Thread 7 in [1]. There are only few reports from Fx85, but none show NSS waiting on any locks.

In [1], NSS is waiting on exactly the same lock that is addressed by the patch in [2].

@RyanVM, I can cut a point release of NSS 3.59 if/when requested.

[1] https://crash-stats.mozilla.org/report/index/6d464f19-d0f7-43cc-9f52-b3ee00201216#allthreads
[2] https://hg.mozilla.org/projects/nss/rev/19585ccc7a1f0f4e9a8d2b9c5ceeb408ea90acb9

Assignee: nobody → kjacobs.bugzilla
Status: UNCONFIRMED → ASSIGNED
Component: Libraries → Security: PSM
Ever confirmed: true
Product: NSS → Core
Version: other → unspecified

Our customers are also affected by this problem with Firefox 84, both under Windows 10 (with bit4xpki.dll) and under macOS Big Sur 11.1 (libbit4xpki.dylib). In both cases, using Firefox BETA solves the problem.
I do not reproduce, instead, with my Kubuntu 20.04 using Firefox 84.0, but I use opensc here.

FF85 beta3 doesn't seem to resolve the issue, at least not on all card types.
On oberthur italian CRS cards it hangs, while with the same configurations it almost works on Atherna smart cards,

Tested on windows 10 64 bits.

I'm sorry to say that FF 83 seems to be affected on Fedora 32 64bit.
The problem arised (yesterday) after the upgrade of NSS to 3.59. Downgrading it to 3.58 (3.58.0-3.fc32 to be precise) is a stable workaround.

(In reply to romolo.manfredini from comment #11)

On oberthur italian CRS cards it hangs, while with the same configurations it almost works on Atherna smart cards,

Can you better specify what you mean with "almost works"?
Still have some problem in Fx85 beta3?

(In reply to Kevin Jacobs [:kjacobs] from comment #7)

  1. From the data on crash-stats, a high volume of the Fx84 reports show NSS waiting as in Thread 7 in [1]. There are only few reports from Fx85, but none show NSS waiting on any locks.

Since in my case Ffx85 beta3 freeze but doesn't crash, there is a way to force a crash report so I can show you where it hangs?

Bug 1683004 has been landed for Firefox 84.0.1, which bumps the minimum NSS version to 3.59.1 and should resolve the crashes for Firefox 84 users when it ships tomorrow. From a tracking standpoint, it would be better if we could take the hangs to a new bug so we can track one issue per bug (especially since those seem to affect all versions and not just 84).
https://hg.mozilla.org/releases/mozilla-release/rev/bf27056390b7

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox84: affected → fixed
Resolution: --- → FIXED

Added to the Firefox 84.0.1 relnotes:

Fixed problems loading secure websites and crashes for users with certain third-party PKCS11 modules and smartcards installed

relnote-firefox: --- → 84+

I tried reproducing the crash using a Yubikey 4 and Yubikey 4 neo (only smartcards I have laying around) and OpenSC on Firefox 84, but without success. Not sure If I can use other PKCS#11 modules, if it is possible can you help with some details of how to do that?
Else can you please verify if this is indeed fixed using 84.0.1 on your machine and with your setup, thanks?
https://archive.mozilla.org/pub/firefox/candidates/84.0.1-candidates/build1/

Flags: needinfo?(midori3)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #17)

I tried reproducing the crash using a Yubikey 4 and Yubikey 4 neo (only smartcards I have laying around) and OpenSC on Firefox 84, but without success. Not sure If I can use other PKCS#11 modules, if it is possible can you help with some details of how to do that?
Else can you please verify if this is indeed fixed using 84.0.1 on your machine and with your setup, thanks?
https://archive.mozilla.org/pub/firefox/candidates/84.0.1-candidates/build1/

I can reproduce the freeze also with 84.0.1-candidates/build1
I'm not sure it can be easily reproduced with an USB Token instead of a separate smart card + reader, since when you unplug the token you also remove the device that the OS see as a smart card reader and this may change a lot the behavior of the underlying PKCS#11 MW.

Since the freeze is caused by a race condition, it depends mainly from the response speed of the device/PKCS#11 and there are cases where this never happens, especially if the PKCS#11 responds really fast.

Using following procedure 100% reproduces the freeze for me with various PKCS#11 MW:

  1. configure PKCS#11 modules (remove osclient lib if present and add MW DLL)
  2. restart Fx
  3. go to https sites (twitter, fb, corriere.it, repubblica.it and some more)
  4. remove card and reload some site
  5. insert card and reload some site
  6. go to HTTPS client auth site: https://server.cryptomix.com/secure/
  7. start over from step 3 until freeze or HTTPS problems
    7.1) alternatively restart Firefox and start over

Usually it requires only 1 iteration with many PKCS#11 MW and never more that 5.
Normally it freeze at step 4, about 2~4 seconds after reinsert the card.

Flags: needinfo?(midori3)

(In reply to J.K.Umeboshi from comment #18)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #17)

I tried reproducing the crash using a Yubikey 4 and Yubikey 4 neo (only smartcards I have laying around) and OpenSC on Firefox 84, but without success. Not sure If I can use other PKCS#11 modules, if it is possible can you help with some details of how to do that?
Else can you please verify if this is indeed fixed using 84.0.1 on your machine and with your setup, thanks?
https://archive.mozilla.org/pub/firefox/candidates/84.0.1-candidates/build1/

I can reproduce the freeze also with 84.0.1-candidates/build1
I'm not sure it can be easily reproduced with an USB Token instead of a separate smart card + reader, since when you unplug the token you also remove the device that the OS see as a smart card reader and this may change a lot the behavior of the underlying PKCS#11 MW.

Since the freeze is caused by a race condition, it depends mainly from the response speed of the device/PKCS#11 and there are cases where this never happens, especially if the PKCS#11 responds really fast.

Using following procedure 100% reproduces the freeze for me with various PKCS#11 MW:

  1. configure PKCS#11 modules (remove osclient lib if present and add MW DLL)
  2. restart Fx
  3. go to https sites (twitter, fb, corriere.it, repubblica.it and some more)
  4. remove card and reload some site
  5. insert card and reload some site
  6. go to HTTPS client auth site: https://server.cryptomix.com/secure/
  7. start over from step 3 until freeze or HTTPS problems
    7.1) alternatively restart Firefox and start over

Usually it requires only 1 iteration with many PKCS#11 MW and never more that 5.
Normally it freeze at step 4, about 2~4 seconds after reinsert the card.

Thanks for the info, I did manage to reproduce the crash using Firefox 84.0:

  1. Windows 7
  1. Ubuntu 18.04

I think the freeze issue will be dealt in another bug and this covers only the crash if I understand correctly. I will go ahead and mark it as verified for not getting the crash anymore.

Status: RESOLVED → VERIFIED
status-firefox84: fixed → verified

I've filed bug 1683891 to track the ongoing hangs/freezes which are happening on 84+ still.

version 84.0.1 didn't resolve the issue with the PKCS#11 when token is to be downloaded from sercure key (in my case Gemalto). Certificate fails to load every time. In Edge it works ok.

For anyone still impacted, you can try enabling osclientcerts rather than loading the module directly into Firefox (see: https://blog.mozilla.org/security/2020/04/14/expanding-client-certificates-in-firefox-75/).

I'm going to revert this patch in NSS back to the NSS 3.58 (Fx83) version. This will reopen bug 1663661, but avoid further hang issues.

thank you, that solved my problem

Keywords: regression
Regressed by: 1663661
Has Regression Range: --- → yes

(In reply to Kevin Jacobs [:kjacobs] from comment #22)

For anyone still impacted, you can try enabling osclientcerts rather than loading the module directly into Firefox (see: https://blog.mozilla.org/security/2020/04/14/expanding-client-certificates-in-firefox-75/).

I'm going to revert this patch in NSS back to the NSS 3.58 (Fx83) version. This will reopen bug 1663661, but avoid further hang issues.

I'm here because i was redirected from a bug i filed for the same issue

The links says

Availability

This library is shipping as part of Firefox Desktop on Windows and macOS, starting with version 75. To enable it, set the about:config preference “security.osclientcerts.autoload” to true.

For users running various flavors of Linux, the OpenSC project (https://github.com/OpenSC/OpenSC/wiki) can provide similar functionality.

I'm using OpenSC, the smartcard seems to be loaded inside Settings > Privacy and Security but not www.inps.it nor any italian e-gov site with CNS access can read the card like it's missing from the reader

You need to log in before you can comment on or make changes to this bug.