Closed Bug 1337950 Opened 7 years ago Closed 7 years ago

Cannot enable FIPS in Firefox 53.0a2

Categories

(Core :: Security: PSM, defect, P1)

Product:
Core
Component:
Security: PSM
Version:
53 Branch
Platform:
x86
Windows 10
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla55
Tracking Flags:
Tracking Status
firefox-esr52 --- unaffected
firefox53 + wontfix
firefox54 + verified
firefox55 + verified

People

(Reporter: nONoNonO, Assigned: keeler, NeedInfo)

References

(Blocks 1 open bug)

Details

(Keywords: dataloss, Whiteboard: [psm-assigned])

Attachments

(1 file)

bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization
59 bytes, text/x-review-board-request
Cykesiopka
: review+
jcj
: review+
gchang
: approval-mozilla-beta+
Details 
After starting Firefox 53.0a2 Developer Edition and creating a new profile, setting a master password under Tools -> Options ->Security, it is not possible to enable FIPS mode under Tools -> Options -> Advanced -> Security Devices.

I get an error message 'Unable to change the FIPS mode for the security device. It is recommended that you exit and restart this application.'

Exiting Firefox and restarting unfortunately doesn't help.

There is no message in the Web Console.
Blocks: fips
Product: Firefox → Core
With mozregression I chased this bag to bug 1295937
s/bag/back/ :-)

2017-02-09T19:43:11: INFO : Narrowed inbound regression window from [95e32667, c138e567] (3 revisions) to [95e32667, ee707767] (2 revisions) (~1 steps left)
2017-02-09T19:43:11: DEBUG : Starting merge handling...
2017-02-09T19:43:11: DEBUG : Using url: https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=ee70776759bf296c951a9805d5b79169291be4d3&full=1
2017-02-09T19:43:13: DEBUG : Found commit message:
bug 1295937 - build NSS using gyp files. r=glandium

MozReview-Commit-ID: Gm1PLWSJwbD

2017-02-09T19:43:13: INFO : The bisection is done.
2017-02-09T19:43:13: INFO : Stopped
It's likely that the new build system doesn't actually support some parts of the NSS fips mode. The question is if it should.
Flags: needinfo?(ted)
Flags: needinfo?(rlb)
I'm afraid I don't know much about FIPS mode aside from the fact that it requires the .chk files present to operate. Are there some other build settings that impact FIPS?

Historically we've been sort of tepid on FIPS support in Firefox--it's not like our Firefox builds are actually FIPS certified, so enabling FIPS mode in a Mozilla-provided Firefox build doesn't seem all that useful.
Flags: needinfo?(ted)
Thinking about this again I don't think Firefox can offer a FIPS NSS token at the moment (not using a system NSS) as we disabled certain parts of NSS in Firefox that would be necessary for FIPS.
I guess NSS_FORCE_FIPS is the make flag (which didn't get ported to gyp).
I don't think we ever used NSS_FORCE_FIPS to build NSS previously, though:
https://hg.mozilla.org/mozilla-central/annotate/2b31faccc67c/config/external/nss/Makefile.in
So, what happens next? Leave FIPS in Firefox broken? Fix it? Or strip it from Firefox? I only use it for testing to find bugs like these, but I suppose it's used in a couple of big enterprises as a company policy. Maybe you have some metrics about FIPS-usage?
A better error message would be good I guess. But if you have a FIPS certified token and want to use that I don't see a reason why it shouldn't work. It just won't work with the default NSS version you get with Firefox (or any version of NSS for that matter as Firefox requires NSS versions that are newer than the most up to date FIPS certified NSS version).
This also happens on macOS (10.12.3) and Firefox 51.0.1.

This time, I am trying to activate the FIPS on Software Security, after adding a master password via about:preferences#advanced , clicked on "Certificate" header tab, then "Security Devices" button.

In the new Device Manager window, left navigation panel, click on the "Software Security Device".  (This step may be extraneous but included here for accuracy).

Then click on "Enable FIPS" button.

An "ALERT" popup dialog box shows with a description of "Unable to change the FIPS mode for the security device. It is recommended that you exit and restart this application."

Restarting the app
(In reply to S. Egbert from comment #9)
> This also happens on macOS (10.12.3) and Firefox 51.0.1.

FYI we broke FIPS mode in Firefox 34 on OS X due to Apple's code signing requirements for app bundles (bug 1047584). We can't have extra files next to the application binaries that weren't there when code signing was performed, and code signing modifies the binaries, so we can't run shlibsign after running codesign, nor can we run shlibsign before running codesign.
Redirecting to Selena / EKR for the policy question.

Personally, ISTM we need to either make FIPS mode actually work or change Firefox to stop pretending it has FIPS mode.
Flags: needinfo?(sdeckelmann)
Flags: needinfo?(rlb)
Flags: needinfo?(ekr)
FWIW, FIPS has also caused in-product problems, notably with Sync (e.g. bug 443386 -- markh's last comment there (#34) indicates this was still a problem as of a couple months ago). Overdue for removal, IMO.
Blocks: 1357436
If you upgrade from FX52 with FIPS enabled (or older) to FX53, you end up with disabled FIPS, no master password and all your saved passwords inaccessible...
[Tracking Requested - why for this release]:


Because FIPS is broken users who upgrade from 52 with FIPS enabled are seeing their signons database broken and are unable to get at their saved passwords without downgrading.
tracking-firefox53: --- → ?
tracking-firefox54: --- → ?
tracking-firefox55: --- → ?
Having users suddenly "lose" their passwords sounds like a critical issue.

Does this need escalated?
It's possible the loss of passwords is caused by Firefox attempting to initialize an NSS DB that is marked as "FIPS", failing, and then falling back to "no DB" mode, which of course wouldn't have access to the key that encrypted the passwords. I wrote a quick test program to try this out, and it looks like if we remove the user's secmod.db (realistically, we would just rename it something like "secmod.db.fips"), Firefox should succeed in loading the key DB, even if the DB was in FIPS mode (since that configuration is stored in the module DB). If that succeeds, the user will have a new non-FIPS secmod.db and we can move forward without this brokenness.
it happens on firefox 55 nightly too.
Tracking for 53 since this sounds like bad data loss newly broken in 53. Is this something we think has a broad user impact?
status-firefox53: --- → affected
status-firefox54: --- → affected
status-firefox55: --- → affected
tracking-firefox53: ? → +
tracking-firefox54: ? → +
tracking-firefox55: ? → +
David, can we tell who uses this from telemetry? It sounds like a non-default configuration.
Flags: needinfo?(dkeeler)
In release, it looks like of those who have enabled telemetry, 0.01% have FIPS enabled: https://mzl.la/2pZxfIT
That said, my understanding is that FIPS is a requirement in US government usage, which is a population that might not commonly enable telemetry.
Flags: needinfo?(dkeeler)
Assignee: nobody → dkeeler
Component: Security → Security: PSM
Priority: -- → P1
Whiteboard: [psm-assigned]
Flags: needinfo?(sdeckelmann)
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

https://reviewboard.mozilla.org/r/133176/#review136498

::: security/certverifier/NSSCertDBTrustDomain.cpp:1231
(Diff revision 1)
>    }
>    if (!loadPKCS11Modules) {
>      flags |= NSS_INIT_NOMODDB;
>    }
> +  MOZ_LOG(gCertVerifierLog, LogLevel::Debug,
> +          ("InitializeNSS(%s,  %d, %d)", dir, readOnly, loadPKCS11Modules));

Nit: extra space there after `%s,  `

::: security/manager/ssl/nsNSSComponent.cpp:1757
(Diff revision 1)
> +  if (srv == SECSuccess) {
> +    MOZ_LOG(gPIPNSSLog, LogLevel::Debug, ("initialized NSS in r/w mode"));
> +    return NS_OK;
> +  }
> +#ifndef ANDROID
> +  savedPRErrorCode1 = PR_GetError();

nit: Since there's no new block here, it seems to me unnecessary to split the declarations from the assignments of `savedPRErrorCode1` and `...2`, but.. eh.

::: security/manager/ssl/nsNSSComponent.cpp:1866
(Diff revision 1)
>    MOZ_LOG(gPIPNSSLog, LogLevel::Debug, ("inSafeMode: %u\n", inSafeMode));
>  
> -  PRErrorCode savedPRErrorCode1 = 0;
> -  PRErrorCode savedPRErrorCode2 = 0;
> -
> -  if (!nocertdb && !profileStr.IsEmpty()) {
> +  rv = InitializeNSSWithFallbacks(profileStr, nocertdb, inSafeMode);
> +  if (NS_FAILED(rv)) {
> +    MOZ_LOG(gPIPNSSLog, LogLevel::Debug, ("failed to initialize NSS"));
> +    return rv;

So previously if the nodb form still failed, we'd `MOZ_CRASH`; this code doesn't. I imagine in general that we want to have fewer `MOZ_CRASH`es, but I want to be sure that this was a deliberate change.
Attachment #8861211 - Flags: review?(jjones) → review+
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

https://reviewboard.mozilla.org/r/133176/#review137344

Cool.

::: security/manager/ssl/nsNSSComponent.cpp:1785
(Diff revision 1)
> +    // It would make sense to initialize NSS in read-only mode here since this
> +    // is just a test to see if the PKCS#11 module DB being in FIPS mode is the
> +    // problem, but for some reason the combination of read-only and no-moddb
> +    // flags causes NSS initialization to fail, so unfortunately we have to use
> +    // read-write mode.
> +    srv = ::mozilla::psm::InitializeNSS(profilePathCStr, false, false);

Something for a follow-up bug, but ugh, this function should take an nsACString and two enums or something instead.

::: security/manager/ssl/tests/unit/test_broken_fips.js:40
(Diff revision 1)
> -    equal(decrypted, testcase.plaintext,
> -          "decrypted ciphertext should match expected plaintext");
> +        "decrypted ciphertext should match expected plaintext");
> +
> +  let secmodDBFileFIPS = do_get_profile();
> +  secmodDBFileFIPS.append(`${secmodDBName}.fips`);
> +  ok(secmodDBFileFIPS.exists(), "backed-up PKCS#11 module db should now exist");

I was originally going to suggest that we test the opposite on Linux, but it turns out Firefox on my Linux machine can't use FIPS and hits the same workaround, so nevermind.
Attachment #8861211 - Flags: review?(cykesiopka.bmo) → review+
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

https://reviewboard.mozilla.org/r/133176/#review136498

Thanks!

> Nit: extra space there after `%s,  `

Fixed.

> nit: Since there's no new block here, it seems to me unnecessary to split the declarations from the assignments of `savedPRErrorCode1` and `...2`, but.. eh.

I think I originally did that because of the comment. I rearranged things so there's less preprocessor junk.

> So previously if the nodb form still failed, we'd `MOZ_CRASH`; this code doesn't. I imagine in general that we want to have fewer `MOZ_CRASH`es, but I want to be sure that this was a deliberate change.

That was a diagnostic crash from bug 1301407. Since that was fixed (in bug 1339267) I don't think we need it any longer.
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

https://reviewboard.mozilla.org/r/133176/#review137344

Thanks!

> Something for a follow-up bug, but ugh, this function should take an nsACString and two enums or something instead.

Yeah, good call. Filed bug 1360307.

> I was originally going to suggest that we test the opposite on Linux, but it turns out Firefox on my Linux machine can't use FIPS and hits the same workaround, so nevermind.

In the past, the *.chk files NSS needed to load in FIPS mode were in a different directory from the libraries when built with ./mach build, and the loading code didn't handle this. Right now, though, it doesn't even look like we're generating those files unless ./mach package is run. In both cases, I think making a packaged build would make it work. In any case, I didn't want to write a test that would only succeed on the test infrastructure (which does use packaged builds, if I understand correctly), so I just opted to disable it.
Try looks good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e7fade1803a8 (not sure what's holding up the OS X tests, though...)
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9daa585e4e9a
work around failing to load a FIPS PKCS#11 module DB in NSS initialization r=Cykesiopka,jcj
https://hg.mozilla.org/mozilla-central/rev/9daa585e4e9a
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox55: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
I've seen additional reports of users (mostly on reddit where I help, but also sumo) who have lost their passwords.
Are we able to do anything for release users to mitigate this?
I'm guessing will uplift this to 54.
:keeler asked if I could post some additional context about how FIPS-140 is used in government.

Broadly, FIPS-140 compliance can be divided into two buckets:

1) Usage of exclusively FIPS-140 approved algorithms (AES, SHA2, ECDH, etc.)
2) Usage of FIPS-validated (CMVP) cryptographic modules

Different agencies apply these to themselves differently; so despite being a "US Federal government standard", what it means in practice isn't very standardized :-)

In my experience, FIPS-140 (1 or 2) was very frequently enforced for server-side software, and occasionally for desktop software, I've never seen it applied to a browser though.

I reviewed the DISA (Defense Information Systems Agency) STIG (Security Technical Implementation Guide) for Firefox -- this is a document that DISA publishes on how to harden/configure software for secure use, they aren't binding (except maybe within DoD), but other agencies tend to use them as references/voluntarily choose to use them -- the STIG didn't mention our FIPS-140 mode, which further contributes to the hypothesis that it is not widely used, and not worth supporting if there's a maintenance burden.

(I assume Red Hat will continue to support FIPS-140 validation for NSS itself)
Sounds like this could use a Beta nomination at least. Not sure if there's any plans for an Fx53 desktop dot release at this point, though.
Flags: needinfo?(dkeeler)
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

Approval Request Comment
[Feature/Bug causing the regression]: bug 1295937
[User impact if declined]: a small percentage of users will not be able to use saved passwords until they update to 55
[Is this code covered by automated tests?]: yes
[Has the fix been verified in Nightly?]: no
[Needs manual test from QE? If yes, steps to reproduce]: yes

1. Install Firefox 52 on Windows
2. Set a master password
3. enable FIPS: about:preferences -> Advanced -> Certificates -> Security Devices -> Enable FIPS
4. Get the password manager to save some passwords
5. Update to Firefox 53
6. Verify that passwords aren't auto-filled / available
7. Update to Firefox 55
8. Passwords should be available again
9. There should be a new "secmod.db.fips" file in the profile directory

[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: not very
[Why is the change risky/not risky?]: it's a bit of a hack, but it's as defensive as possible and it should only affect an extremely small percentage of users
[String changes made/needed]: none
Flags: needinfo?(dkeeler)
Attachment #8861211 - Flags: approval-mozilla-beta?
Hi Brindusa, could you help find someone to verify if this issue was fixed as expected on a latest Nightly build? Thanks!
Flags: qe-verify+
Flags: needinfo?(brindusa.tot)
Verified bug on Windows 8.1 x64:

1. installed 52.0 20170302120751
5. updated to 54.0b4 20170501133120 - master password/passwords not available
7-8. updated to 55.0a1 20170504030320 - master password / passwords autofill available.
9. secmod.db.fips created in the profile directory.

Updating tracking flag for 55.
status-firefox55: fixed → verified
Flags: needinfo?(brindusa.tot)
Redid the STR using 53.0b9 20170403072723 at step 5. and same behavior. One additional note, on beta 53/54 you cannot enable FIPS. The user story error is thrown: 'Unable to change the FIPS mode for the security device. It is recommended that you exit and restart this application.' 

David, this problem was reported on OSX as well (comment 9). Is this fix affecting OSX as well?
Flags: needinfo?(dkeeler)
Yes, this affects OS X as well. The issue there is that you can't enable FIPS in any version after 34 on OS X, which makes it harder to test (if you can install and run version 33, that would work, but I think the platform signing requirements prevent this, unless I'm misunderstanding). One thing you could do is create a FIPS profile on Windows, transfer it to an OS X machine, and then do the rest of the STR.
Flags: needinfo?(dkeeler)
I'm not sure it's worth testing on OSX, since FIPS mode has been busted there for three years; presumably anyone missing their password database fixed it one way or another since then...
Comment on attachment 8861211 [details]
bug 1337950 - work around failing to load a FIPS PKCS#11 module DB in NSS initialization

Fix a data loss issue. Beta54+. Should be in 54 beta 6.
Attachment #8861211 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
I have reproduced this issue using Firefox 53 on Win 10 x86.
I can confirm this issue is fixed, I verified using Firefox 54.0b8 on Win 10 x86.
Status: RESOLVED → VERIFIED
status-firefox54: fixed → verified
Flags: qe-verify+
Can confirm this has been resolved in Thunderbird 54.0b1 too. (Bug #1357436)
Seems like a wontfix for 53 at this point.
status-firefox53: affected → wontfix
Not sure why you'd need this fixed in 53 when it got fixed for 54. (54 is available now.)
Depends on: 1382994
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: