1494431 - Pin *.firefox.com to mozilla services whitelisted roots in Firefox

Status: RESOLVED INACTIVE

(Cloud Services :: Security, enhancement)

Description

[Julien Vehent]

Attachment: certificates from *.firefox.com not issued by digicert

We are creating a lot of new products and services under *.firefox.com that do not  get added to the list of preloaded hpk pins [1]. We are currently only using Digicert certificates on *.firefox.com (to the exception of design.f.c, issued by LE, see attachment), so I propose that we put the entire *.firefox.com domain in the list of sites covered by the mozilla services whitelisted roots (which currently include Digicert and Let's Encrypt).

This will reduce the risk of a fraudulent CA hijacking Firefox sites and serving malicious content, while also not depending on the somewhat-defunct hpkp.

Thoughts?

[1] https://dxr.mozilla.org/mozilla-central/source/security/manager/tools/PreloadedHPKPins.json

Comment 1

[J.C. Jones [:jcj] (he/they)]

Hmmm... I assume we're good with requiring HTTPS for all of firefox.com, too? Like, we don't have any mandatory-HTTP endpoints there for updaters or some such?

Comment 2

[Julien Vehent]

detectportal.firefox.com is one HTTP site we need to keep before this can happen.

jcj: does adding a site to the preloaded pins mean requiring HTTPS for that site? I assume the answer is yes based on your comment.

Comment 3

[Julien Vehent]

Here's a list of *.firefox.com domains extracted from mozilla central:
     41 https://accounts.firefox.com
     14 https://screenshots.firefox.com
     12 http://design.firefox.com
     12 http://www.firefox.com
      6 https://profile.accounts.firefox.com
      4 https://oauth.accounts.firefox.com
      4 https://testpilot.firefox.com
      3 https://api.accounts.firefox.com
      3 https://marketplace.firefox.com:443
      2 http://detectportal.firefox.com
      1 http://hidden.firefox.com
      1 http://self.firefox.com
      1 https://verifier.accounts.firefox.com

I don't know what "hidden" and "self" are. I suspect they are tests since the records don't exists in DNS.
"design" supports and redirects to HTTPS, so no problem there.

Comment 4

[J.C. Jones [:jcj] (he/they)]

(In reply to Julien Vehent [:ulfr] from comment #2)
> detectportal.firefox.com is one HTTP site we need to keep before this can
> happen.
> 
> jcj: does adding a site to the preloaded pins mean requiring HTTPS for that
> site? I assume the answer is yes based on your comment.

I'll get a double-check from Dana, but I believe so: Adding a site to the key pins will make it 'true' for PublicKeyPinningService::HostHasPins, which is called by nsSiteSecurityService::IsSecureURI as part of page construction. I believe that will require HTTPS, but I'm not totally certain. (Could always just make a build and try it, I suppose).

Dana?

Comment 5

[Dana Keeler (she/her) [:keeler]]

Pinned sites aren't upgraded to https, so we shouldn't need the carveout for http://detectportal.firefox.com (as an aside, presumably we're doing a good job protecting other *.firefox.com cookies from attackers injecting code into that http-only site?)
Pinned sites aren't allowed to have certificate error overrides, though (which, incidentally, means our error page is a bit misleading for pinned, non-HSTS domains, since it only mentions HSTS).

Comment 6

[J.C. Jones [:jcj] (he/they)]

Comment on attachment 9012329 [details]
certificates from *.firefox.com not issued by digicert

OK, so it sounds good to me. When do you want to make this change?

Comment 7

[Julien Vehent]

I'll submit a patch now to run it in test mode for at least one release cycle, and if telemetry is happy, we can switch it on.

Comment 8

[Julien Vehent]

Attachment: Bug 1494431 - Pin *.firefox.com to mozilla services whitelisted roots

Put the entire *.firefox.com domain in the list of sites covered by the mozilla services whitelisted roots, which currently include Digicert and Let's Encrypt.

Comment 9

[Dana Keeler (she/her) [:keeler]]

Comment on attachment 9013004 [details]
Bug 1494431 - Pin *.firefox.com to mozilla services whitelisted roots

Dana Keeler [:keeler] (she/her) (use needinfo) has approved the revision.

Comment 10

[J.C. Jones [:jcj] (he/they)]

Comment on attachment 9013004 [details]
Bug 1494431 - Pin *.firefox.com to mozilla services whitelisted roots

J.C. Jones [:jcj] (he/him) has approved the revision.

Comment 11

[Julien Vehent]

Alright, let's try this on nightly.

:keeler, do you remember how we would track this in telemetry?

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/351f159a3fd8
Pin *.firefox.com to mozilla services whitelisted roots r=keeler,jcj

Comment 13

[Dana Keeler (she/her) [:keeler]]

I think we'd be looking at the CERT_PINNING_MOZ_TEST_RESULTS_BY_HOST histogram. The buckets are (2 * id) + (success ? 1 : 0)
( https://searchfox.org/mozilla-central/rev/819cd31a93fd50b7167979607371878c4d6f18e8/security/manager/ssl/PublicKeyPinningService.cpp#300 )

Comment 14

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/351f159a3fd8

Comment 15

[Julien Vehent]

Dana, two questions:
1) Where can I find the compiled list of preloaded pins in a Firefox installation?
2) Would you have a link to a telemetry dashboard where we can track violations for *.firefox.com (15)?

Comment 16

[Dana Keeler (she/her) [:keeler]]

The compiled pin list is https://searchfox.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h but it hasn't been updated by the automated script to include these changes yet (this is the most recent run: https://hg.mozilla.org/mozilla-central/rev/c69fc771bb00 ). Looks like we're running it once every 3 days or so.

Here's a link to a dashboard: https://mzl.la/2zMfuke
Bucket 30 will be failures and bucket 31 will be successes. Right now, we have mostly success for aus5.mozilla.org and some failures, and only successes for telemetry.mozilla.org.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Do we need to backport this to Beta and/or ESR60?

Comment 18

[Julien Vehent]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> Do we need to backport this to Beta and/or ESR60?

We do not. It is still in test mode (not effectively blocking) and will gradually roll out to future releases.

Comment 19

[Julien Vehent]

Quoting :keeler on telemetry analysis:

> on beta it looks like 1.46k failures compared to 187k successes

That's a 0.8% failure rate, which is quite high. We probably need to investigate it further before turning it on.

Comment 20

[Dana Keeler (she/her) [:keeler]]

What's odd is the failures are disproportionately from windows7 installations.

Comment 21

[Julien Vehent]

I suggest we let this ride in testing mode to release with 64, then use telemetry to decide if we want to flip it in 66.

Comment 22

[Julien Vehent]

According to [1], the probe is pre-release only, so we're not getting any telemetry for the release channel. Dana, is this something you'd like to change?

Meanwhile, looking at the last 30 days for nightly 66 and dev edition 65, I count:

• 18,756 (6 + 18,750) entries in bucket 30 (failures)
• 749,480 (215,400 + 534,080) entries in bucket 31 (successes)

That's a failure rate of 2.5%, concentrated on dev edition 65 on windows 10. The data there is actually worse, with 18,750 failures and 338,770 successes, or a failure rate of 5.5%. This is too high up to enable enforcement, and we need more data to figure out what is happening here.

[1] https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_test_results_by_host&searchtype=in_name&channel=nightly

Comment 23

[Dana Keeler (she/her) [:keeler]]

(In reply to Julien Vehent [:ulfr] from comment #22)

According to [1], the probe is pre-release only, so we're not getting any telemetry for the release channel. Dana, is this something you'd like to change?

Yeah, that might be a good idea.

Meanwhile, looking at the last 30 days for nightly 66 and dev edition 65, I count:

• 18,756 (6 + 18,750) entries in bucket 30 (failures)
• 749,480 (215,400 + 534,080) entries in bucket 31 (successes)

That's a failure rate of 2.5%, concentrated on dev edition 65 on windows 10. The data there is actually worse, with 18,750 failures and 338,770 successes, or a failure rate of 5.5%. This is too high up to enable enforcement, and we need more data to figure out what is happening here.

That seems quite high. Some ideas:

• it could be that a significant portion of those users have changed their pinning settings from "allow mitm" (the default) to "strict" (particularly since these are pre-release/dev users) (maybe we should only gather telemetry if the pref is set to the default?) (these users would also have to be behind some intercepting device/software)
• small measurements/populations could be throwing this off. In beta 65, I'm seeing 13K failures vs. 1.4M successes (which I guess is still about 1%, and is higher than the other test sites we're measuring for)
• maybe we have some forgotten foo.firefox.com domain that doesn't use the right CA
• maybe someone is intercepting traffic to firefox.com with a cert from a publicly-trusted CA (I think this is unlikely)

[1] https://telemetry.mozilla.org/probe-dictionary/?search=cert_pinning_moz_test_results_by_host&searchtype=in_name&channel=nightly

Comment 24

[Julien Vehent]

I'm also concerned about the small size of the sample. Enabling the probe in release seems important before we can take this any further. Should that be a separate bug?