1030135 - enable pinning on services.mozilla.com

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Frederik Braun [:freddy]]

+++ This bug was initially created as a clone of Bug #1020485 +++

I suggest we enable certificate pinning for the loop servers.

As per bug 1020485 comment 5 and https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning/SiteOperators this would require the loop server to give notice 14 weeks before changing your issuer CA (including CDNs and optionally subdomains).

Adam, Ben: Are you the right people to talk to about this? If not, can you please assign and CC the relevant people?

Comment 1

[Maire Reavy [:mreavy]]

Frederik -- What changes do we need for this?  Does this block v2.0?

Comment 2

[Frederik Braun [:freddy]]

This should probably block loop, but not loop-mobile-client. No v2.0 blocker.

Comment 3

[Maire Reavy [:mreavy]]

Should we add this to the v2.1 backlog?

Comment 4

[Frederik Braun [:freddy]]

Nope. As Monica explained in bug 1020485 comment 27, it doesn't make a lot of sense in b2g. But it's still desirable for Loop on Firefox desktop.

Changing blocker/depending fields accordingly.

Comment 5

[Maire Reavy [:mreavy]]

Thanks for clarification.  That makes much more sense.

Comment 6

[[:mmc] Monica Chew (no longer reading bugmail)]

Can we expand this to all of *.services.mozilla.com? Freddy, can you coordinate with cloud services team to figure this out?

Comment 7

[Ed Lee :Mardak]

*.services.mozilla.com would include at least tracking protection, tiles, and sync (from scanning about:config).

Comment 8

[[:mmc] Monica Chew (no longer reading bugmail)]

Hey ckolos,

I remember that you helped out with the FxA bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1020485). Could you provide us a similar list for services.mozilla.com, or redirect this bug to someone who can?

Thanks,
Monica

Comment 9

[Benjamin Smedberg]

[Tracking Requested - why for this release]: I very much would like the tiles ping to be pin-protected in 33.

Comment 10

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:mmc I would think that we would not recommend that *.services.m.c be pinned only because we regularly add and delete things from it that domain for various reasons, including the development of products that ultimately do nothing/go nowhere.

There are a number of certs currently under the umbrella of s.m.c, including:

*.services.mozilla.com
*.sync.services.mozilla.com
loads.services.mozilla.com
location.services.mozilla.com
loop.services.mozilla.com
presence.services.mozilla.com
push.services.mozilla.com
tracking.services.mozilla.com

As I said, this list can and does change.

Comment 11

[[:mmc] Monica Chew (no longer reading bugmail)]

OK, in that case, could we start out with subdomains that are known to be reaching release soonish, or already in production?

sync.services.mozilla.com
location.services.mozilla.com
loop.services.mozilla.com
tiles.services.mozilla.com

Comment 12

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:mmc  I'd rather the wildcard not get pinned, but if we have to, then I suppose we have to, but it *feels* like bad practice to do so. As it turns out, we have a mish-mosh of "stuff" out there, some individually cert'd and others using the wildcard. I'll need to clunk heads with the team and see if we can hammer out a strategy for this.

That being said, the list I pasted in c10 is a good place to start for individual certs. Tiles is not yet cert'd and is one of the props using the wildcard. Shavar is tracking.s.m.c.

Comment 13

[[:mmc] Monica Chew (no longer reading bugmail)]

Hey ckolos, no need to pin *.s.m.c if you don't think it is a good idea. All we need are the certs that are in use (including the wildcarded cert). We can be very liberal with the pin list.

Comment 14

[[:mmc] Monica Chew (no longer reading bugmail)]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #13)
> Hey ckolos, no need to pin *.s.m.c if you don't think it is a good idea. All
> we need are the certs that are in use (including the wildcarded cert). We
> can be very liberal with the pin list.

Meaning that the list of acceptable certs can be large and only apply to subdomains of s.m.c you think would be safe.

Comment 15

[Daniel Veditz [:dveditz]]

In case it's not clear, we have generally not pinned the specific machine key since those expire and change. Rather most of our sites have been pinned to a Certificate Authority root (or two). Doesn't help if the CA we use gets hacked but eliminates the risk from a hundred other CAs (and gives us incentive to pick our CA for quality rather than on cost alone).

Comment 16

[Frederik Braun [:freddy]]

Seems like this is already taken care of by ckolos. Clearing needinfo.

Comment 17

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:dveditz :mmc - Ah, yes, I forgot that it has more to do with the CA than it does with the actual cert itself. We are using Digicert (exclusively?) for our certs so it shouldn't be a problem to pin against the same CA as we did for accounts.firefox.com

Comment 18

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Enable pinning on services.mozilla.com in test mode (

Comment 19

[[:mmc] Monica Chew (no longer reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/54d37aeae2ce

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/54d37aeae2ce

Comment 21

[[:mmc] Monica Chew (no longer reading bugmail)]

Huh, the approval comments form didn't come up. I am requesting uplift because bsmedberg requested to get this into 33. Specific branch patches will be needed because of buildbot integration.

Approval Request Comment
[Feature/regressing bug #]: None.

[User impact if declined]: services.mozilla.org won't be pinned until Firefox 35 (http://monica-at-mozilla.blogspot.com/2014/08/firefox-32-supports-public-key-pinning.html)

[Describe test coverage new/current, TBPL]: telemetry dashboard at people.mozilla.org/~mchew/pinning_dashboard

[Risks and why]: 
This is to enable pinning in test mode, so it only collects telemetry. It has no other effect. Any follow up changes to enable pinning for real has higher risk.

[String/UUID change made/needed]: None.

Comment 22

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #21)
> Huh, the approval comments form didn't come up. I am requesting uplift
> because bsmedberg requested to get this into 33. Specific branch patches
> will be needed because of buildbot integration.

Please create the branch specific patches and request approval on those when they're ready. I'm going to deny your current patches with the expectation that we need different patches for both aurora and beta.

> Approval Request Comment
> [Feature/regressing bug #]: None.
> 
> [User impact if declined]: services.mozilla.org won't be pinned until
> Firefox 35
> (http://monica-at-mozilla.blogspot.com/2014/08/firefox-32-supports-public-
> key-pinning.html)

After speaking with Monica and Benjamin, the beta request is to pin the tiles server. As this is a subset of services.mozilla.org, can we restrict pinning to that domain in a separate bug for 33?

> [Risks and why]: 
> This is to enable pinning in test mode, so it only collects telemetry. It
> has no other effect. Any follow up changes to enable pinning for real has
> higher risk.

This makes me fairly uncomfortable as pinning for real likely won't happen until beta7. I will be somewhat more comfortable if we restrict pinning to tiles but I think we need to weight the risk vs reward of doing this in 33 before proceeding. I have asked Benjamin to speak with Monica about the points in this response.

Comment 23

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8485100 [details] [diff] [review]
Enable pinning on services.mozilla.com in test mode (

I'm clearing the beta and aurora approval noms as Monica tells me that branch specific patches are needed. Please request approval on the branch specific patches when ready.

Given that this change needs ~2 weeks of measurement time, I would highly prefer to take this change in beta3, which goes to build tomorrow (Thursday).

Comment 24

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Aurora patch to enable pinning on services.mozilla.com in test mode (

Comment 25

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8487369 [details] [diff] [review]
Aurora patch to enable pinning on services.mozilla.com in test mode (

Review of attachment 8487369 [details] [diff] [review]:
-----------------------------------------------------------------

See comment 21.

Comment 26

[Dana Keeler (she/her) (use needinfo) [:keeler]]

I don't think we should make these changes in beta. Even after setting test-only pins, waiting for telemetry to catch up, and then promoting the test pins to production, there's still a considerable chance that we'll break something. We don't want to do that in a late beta.

Comment 27

[[:mmc] Monica Chew (no longer reading bugmail)]

I agree with keeler and am requesting approval for 34 only.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/a91c5b4e960d

Comment 29

[[:mmc] Monica Chew (no longer reading bugmail)]

Re-opening until everything's in production mode.

Comment 30

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Promote services.mozilla.com to production mode (

Comment 31

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Set the bucket for services.mozilla.com (

Comment 32

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Set is_moz if the pinset name contains mozilla (

Comment 33

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8499161 [details] [diff] [review]
Set is_moz if the pinset name contains mozilla (

Review of attachment 8499161 [details] [diff] [review]:
-----------------------------------------------------------------

The problem with the previous change is that violations for pinning.mozilla.org were being thrown into the "test" bucket which also contains facebook at the moment. This throws them into the "mozilla test" bucket instead. I thought about setting the bucket id but that seems kind of like cheating, because some but not all subdomains of services.mozilla.com pass the "operationally critical" test.

I didn't bother updating the output file because it's Thursday and m-i is closed.

Comment 34

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Set is_moz if the pinset name contains mozilla, set bucket id for pinsets containing the string mozilla (

Comment 35

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8499204 [details] [diff] [review]
Set is_moz if the pinset name contains mozilla, set bucket id for pinsets containing the string mozilla (

Review of attachment 8499204 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM. Does this change the telemetry in the tests at all? I guess we'll find out :)

Comment 36

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Set is_moz if the pinset name contains mozilla, set bucket id for pinsets containing the string mozilla (

* * *
Bug 1046430: Make cookie behavior the same on beta/release as on aurora/nightly

Comment 37

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Set is_moz if the pinset name contains mozilla, set bucket id for pinsets containing the string mozilla (

Comment 38

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8499246 [details] [diff] [review]
Set is_moz if the pinset name contains mozilla, set bucket id for pinsets containing the string mozilla (

Review of attachment 8499246 [details] [diff] [review]:
-----------------------------------------------------------------

It did affect the unittest, so fixed. Reverting the generated part of the patch and waiting until try is open.

Comment 39

[[:mmc] Monica Chew (no longer reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/086b51f4ba0b

Comment 40

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/086b51f4ba0b

Comment 41

[[:mmc] Monica Chew (no longer reading bugmail)]

FF 35 is not fixed until we promote this pin to production mode. FF 34 is already in beta, and we have an agreement not to make pinning changes in beta. I cleared the b2g flags because pinning is not enabled on b2g.

Comment 42

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Promote pin for services.mozilla.com to production mode (

Comment 43

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8518973 [details] [diff] [review]
Promote pin for services.mozilla.com to production mode (

Review of attachment 8518973 [details] [diff] [review]:
-----------------------------------------------------------------

Violation rates seem about on par with api.accounts.firefox.com. Since this covers services that are also used on startup (e.g., tiles) we expect this rate to be a little higher than normal.

Comment 44

[[:mmc] Monica Chew (no longer reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2db5b82bf663

Comment 45

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2db5b82bf663

Comment 46

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8518973 [details] [diff] [review]
Promote pin for services.mozilla.com to production mode (

Approval Request Comment
[Feature/regressing bug #]: None, new feature.
[User impact if declined]: tiles.services.mozilla.com won't be pinned until 36. bsmedberg particularly wanted Directory Tiles to be pinned.
[Describe test coverage new/current, TBPL]: Already in m-c for a week with no complaints
[Risks and why]: Pretty low risk
[String/UUID change made/needed]: None

Comment 47

[Lukas Blakk [:lsblakk] use ?needinfo]

Updating TM to 36 since that's where it landed on nightly.

Comment 48

[[:mmc] Monica Chew (no longer reading bugmail)]

Thanks, Lukas!

https://hg.mozilla.org/releases/mozilla-aurora/rev/e093e08145b1

Comment 49

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.