Open Bug 1494431 Opened Last year Updated 9 months ago

Pin *.firefox.com to mozilla services whitelisted roots in Firefox

Categories

(Cloud Services :: Security, enhancement)

enhancement
Not set

Tracking

(firefox64 fixed)

REOPENED
Firefox 64
Tracking Status
firefox64 --- fixed

People

(Reporter: ulfr, Assigned: ulfr)

References

(Depends on 1 open bug)

Details

Attachments

(2 files)

We are creating a lot of new products and services under *.firefox.com that do not  get added to the list of preloaded hpk pins [1]. We are currently only using Digicert certificates on *.firefox.com (to the exception of design.f.c, issued by LE, see attachment), so I propose that we put the entire *.firefox.com domain in the list of sites covered by the mozilla services whitelisted roots (which currently include Digicert and Let's Encrypt).

This will reduce the risk of a fraudulent CA hijacking Firefox sites and serving malicious content, while also not depending on the somewhat-defunct hpkp.

Thoughts?

[1] https://dxr.mozilla.org/mozilla-central/source/security/manager/tools/PreloadedHPKPins.json
Attachment #9012329 - Flags: feedback?(oremj)
Attachment #9012329 - Flags: feedback?(jjones)
Summary: Ping *.firefox.com to mozilla services whitelisted roots in Firefox → Pin *.firefox.com to mozilla services whitelisted roots in Firefox
Hmmm... I assume we're good with requiring HTTPS for all of firefox.com, too? Like, we don't have any mandatory-HTTP endpoints there for updaters or some such?
Depends on: 1494664
detectportal.firefox.com is one HTTP site we need to keep before this can happen.

jcj: does adding a site to the preloaded pins mean requiring HTTPS for that site? I assume the answer is yes based on your comment.
Here's a list of *.firefox.com domains extracted from mozilla central:
     41 https://accounts.firefox.com
     14 https://screenshots.firefox.com
     12 http://design.firefox.com
     12 http://www.firefox.com
      6 https://profile.accounts.firefox.com
      4 https://oauth.accounts.firefox.com
      4 https://testpilot.firefox.com
      3 https://api.accounts.firefox.com
      3 https://marketplace.firefox.com:443
      2 http://detectportal.firefox.com
      1 http://hidden.firefox.com
      1 http://self.firefox.com
      1 https://verifier.accounts.firefox.com

I don't know what "hidden" and "self" are. I suspect they are tests since the records don't exists in DNS.
"design" supports and redirects to HTTPS, so no problem there.
(In reply to Julien Vehent [:ulfr] from comment #2)
> detectportal.firefox.com is one HTTP site we need to keep before this can
> happen.
> 
> jcj: does adding a site to the preloaded pins mean requiring HTTPS for that
> site? I assume the answer is yes based on your comment.

I'll get a double-check from Dana, but I believe so: Adding a site to the key pins will make it 'true' for PublicKeyPinningService::HostHasPins, which is called by nsSiteSecurityService::IsSecureURI as part of page construction. I believe that will require HTTPS, but I'm not totally certain. (Could always just make a build and try it, I suppose).

Dana?
Flags: needinfo?(dkeeler)
Pinned sites aren't upgraded to https, so we shouldn't need the carveout for http://detectportal.firefox.com (as an aside, presumably we're doing a good job protecting other *.firefox.com cookies from attackers injecting code into that http-only site?)
Pinned sites aren't allowed to have certificate error overrides, though (which, incidentally, means our error page is a bit misleading for pinned, non-HSTS domains, since it only mentions HSTS).
Flags: needinfo?(dkeeler)
Comment on attachment 9012329 [details]
certificates from *.firefox.com not issued by digicert

OK, so it sounds good to me. When do you want to make this change?
Attachment #9012329 - Flags: feedback?(jjones) → feedback+
I'll submit a patch now to run it in test mode for at least one release cycle, and if telemetry is happy, we can switch it on.
Put the entire *.firefox.com domain in the list of sites covered by the mozilla services whitelisted roots, which currently include Digicert and Let's Encrypt.
Comment on attachment 9013004 [details]
Bug 1494431 - Pin *.firefox.com to mozilla services whitelisted roots

Dana Keeler [:keeler] (she/her) (use needinfo) has approved the revision.
Attachment #9013004 - Flags: review+
Comment on attachment 9013004 [details]
Bug 1494431 - Pin *.firefox.com to mozilla services whitelisted roots

J.C. Jones [:jcj] (he/him) has approved the revision.
Attachment #9013004 - Flags: review+
Alright, let's try this on nightly.

:keeler, do you remember how we would track this in telemetry?
Keywords: checkin-needed
Pushed by ebalazs@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/351f159a3fd8
Pin *.firefox.com to mozilla services whitelisted roots r=keeler,jcj
Keywords: checkin-needed
I think we'd be looking at the CERT_PINNING_MOZ_TEST_RESULTS_BY_HOST histogram. The buckets are (2 * id) + (success ? 1 : 0)
( https://searchfox.org/mozilla-central/rev/819cd31a93fd50b7167979607371878c4d6f18e8/security/manager/ssl/PublicKeyPinningService.cpp#300 )
Attachment #9012329 - Flags: feedback?(oremj) → feedback+
https://hg.mozilla.org/mozilla-central/rev/351f159a3fd8
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → Firefox 64
Dana, two questions:
1) Where can I find the compiled list of preloaded pins in a Firefox installation?
2) Would you have a link to a telemetry dashboard where we can track violations for *.firefox.com (15)?
Flags: needinfo?(dkeeler)
The compiled pin list is https://searchfox.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h but it hasn't been updated by the automated script to include these changes yet (this is the most recent run: https://hg.mozilla.org/mozilla-central/rev/c69fc771bb00 ). Looks like we're running it once every 3 days or so.

Here's a link to a dashboard: https://mzl.la/2zMfuke
Bucket 30 will be failures and bucket 31 will be successes. Right now, we have mostly success for aus5.mozilla.org and some failures, and only successes for telemetry.mozilla.org.
Flags: needinfo?(dkeeler)
Do we need to backport this to Beta and/or ESR60?
Flags: needinfo?(jvehent)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> Do we need to backport this to Beta and/or ESR60?

We do not. It is still in test mode (not effectively blocking) and will gradually roll out to future releases.
Flags: needinfo?(jvehent)
Quoting :keeler on telemetry analysis:

> on beta it looks like 1.46k failures compared to 187k successes

That's a 0.8% failure rate, which is quite high. We probably need to investigate it further before turning it on.
What's odd is the failures are disproportionately from windows7 installations.
I suggest we let this ride in testing mode to release with 64, then use telemetry to decide if we want to flip it in 66.

According to [1], the probe is pre-release only, so we're not getting any telemetry for the release channel. Dana, is this something you'd like to change?

Meanwhile, looking at the last 30 days for nightly 66 and dev edition 65, I count:

  • 18,756 (6 + 18,750) entries in bucket 30 (failures)
  • 749,480 (215,400 + 534,080) entries in bucket 31 (successes)

That's a failure rate of 2.5%, concentrated on dev edition 65 on windows 10. The data there is actually worse, with 18,750 failures and 338,770 successes, or a failure rate of 5.5%. This is too high up to enable enforcement, and we need more data to figure out what is happening here.

[1] https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_test_results_by_host&searchtype=in_name&channel=nightly

Flags: needinfo?(dkeeler)

(In reply to Julien Vehent [:ulfr] from comment #22)

According to [1], the probe is pre-release only, so we're not getting any telemetry for the release channel. Dana, is this something you'd like to change?

Yeah, that might be a good idea.

Meanwhile, looking at the last 30 days for nightly 66 and dev edition 65, I count:

  • 18,756 (6 + 18,750) entries in bucket 30 (failures)
  • 749,480 (215,400 + 534,080) entries in bucket 31 (successes)

That's a failure rate of 2.5%, concentrated on dev edition 65 on windows 10. The data there is actually worse, with 18,750 failures and 338,770 successes, or a failure rate of 5.5%. This is too high up to enable enforcement, and we need more data to figure out what is happening here.

That seems quite high. Some ideas:

  • it could be that a significant portion of those users have changed their pinning settings from "allow mitm" (the default) to "strict" (particularly since these are pre-release/dev users) (maybe we should only gather telemetry if the pref is set to the default?) (these users would also have to be behind some intercepting device/software)
  • small measurements/populations could be throwing this off. In beta 65, I'm seeing 13K failures vs. 1.4M successes (which I guess is still about 1%, and is higher than the other test sites we're measuring for)
  • maybe we have some forgotten foo.firefox.com domain that doesn't use the right CA
  • maybe someone is intercepting traffic to firefox.com with a cert from a publicly-trusted CA (I think this is unlikely)

[1] https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_test_results_by_host&searchtype=in_name&channel=nightly

Flags: needinfo?(dkeeler)

I'm also concerned about the small size of the sample. Enabling the probe in release seems important before we can take this any further. Should that be a separate bug?

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You need to log in before you can comment on or make changes to this bug.