1116409 - switch update server to sha2 cert; update in-tree pinning

Status: VERIFIED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Yuhong Bao]

The intermediates listed in http://lxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js as app.update.certs.* are all SHA1 intermediates.

Comment 1

[Yuhong Bao]

Also affects media.gmp-manager.certs.*.

Comment 2

[[e-mail] (:bhearsum)]

Looks like this probably requires a new certificate to change. That's relatively easy, but needs to be coordinated with IT.

Rob, are we able to transition gracefully to the new one, or will we need to create aus5.mozilla.org to make sure aus4 fingerprints checks don't fail?

Comment 3

[[e-mail] (:bhearsum)]

(For background: https://blog.mozilla.org/security/2014/09/23/phasing-out-certificates-with-sha-1-based-signature-algorithms/)

Comment 4

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> Looks like this probably requires a new certificate to change. That's
> relatively easy, but needs to be coordinated with IT.
> 
> Rob, are we able to transition gracefully to the new one, or will we need to
> create aus5.mozilla.org to make sure aus4 fingerprints checks don't fail?
Just use the same process as we used the last time which is to use a new host name record with the new certs unless the certs have the same attribute values for issuerName and commonName so the clients that point to the old host don't break.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #4)
> (In reply to Ben Hearsum [:bhearsum] from comment #2)
> > Looks like this probably requires a new certificate to change. That's
> > relatively easy, but needs to be coordinated with IT.
> > 
> > Rob, are we able to transition gracefully to the new one, or will we need to
> > create aus5.mozilla.org to make sure aus4 fingerprints checks don't fail?
> Just use the same process as we used the last time which is to use a new
> host name record with the new certs unless the certs have the same attribute
> values for issuerName and commonName so the clients that point to the old
> host don't break.

OK. I'll look into this more soon, but if it involves new hostnames I'd like to wait until after we transition esr to aus4 (or not), just to keep things simple.

Comment 6

[Yuhong Bao]

Well, the old certificates has not expired, so just add the new intermediates for now so when we renew in 2-3 years we will be ready.

Comment 7

[[e-mail] (:bhearsum)]

Joe, any thoughts on how critical this is? Changing the certs + reving the aus hostname isn't something I'm particularly eager to do.

Comment 8

[[e-mail] (:bhearsum)]

Joe, any thoughts on how critical this is? Changing the certs + reving the aus hostname isn't something I'm particularly eager to do.

Comment 9

[Yuhong Bao]

AFAIK, these URLs are only used in Mozilla, so we don't care about MS or Google deadlines relating to SHA1 certificates.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Yuhong Bao from comment #9)
> AFAIK, these URLs are only used in Mozilla, so we don't care about MS or
> Google deadlines relating to SHA1 certificates.

Yup, that's right, but they're being deprecated for security reasons, so if we strongly that this could enable MitM or other attacks, we should prioritize this earlier.

Comment 11

[Joe Sabash [:JoeS1]]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Joe, any thoughts on how critical this is? Changing the certs + reving the
> aus hostname isn't something I'm particularly eager to do.

I guess your needinfo to me was a mistake, but FWIW I haven't heard of any MITM attempts in any of the forums that I participate in.

Comment 12

[Joe Stevensen [:joe]]

Ben, we recommend replacing the certs. Can this be done before the end of the year?

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Joe Stevensen [:joe] from comment #12)
> Ben, we recommend replacing the certs. Can this be done before the end of
> the year?

I don't see why not. It requires some coordination, but it's doable.

Comment 14

[Yuhong Bao]

I think it is possible for a SHA2 cert to be issued off a SHA1 intermediate if Symantec or DigiCert is willing to.

Comment 15

[:Atoll]

Ben, which registrars do we need to provide SHA-2 intermediates for, to help move this forward?

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Richard Soderberg [:atoll] from comment #15)
> Ben, which registrars do we need to provide SHA-2 intermediates for, to help
> move this forward?

I don't have the headspace to look at this for awhile -- probably not until Q3.

Comment 17

[:Atoll]

Np, here any time to help when ready.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Richard Soderberg [:atoll] from comment #15)
> Ben, which registrars do we need to provide SHA-2 intermediates for, to help
> move this forward?

Doesn't matter where we get the certificates from as far as I'm concerned. We just need to update the pinning in https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.js#L152

If it's possible to get one from an intermediary with the same details as the existing DigiCert one, this is even easier. https://www.digicert.com/digicert-root-certificates.htm suggests that SHA2 certs come off of a different intermediary, but I'd be happy to be wrong about that.

Comment 19

[:Atoll]

Precisely which SHA-1 certificates will need to be reissued as SHA-2 to resolve this bug?

Either attaching those certificates, or directing me to the https:// hostname that servers them, will help us proceed here.

Once I have the complete set of certificates that will need to be reissued, I can determine which intermediates will be used for their SHA-2 equivalents.

Comment 20

[Yuhong Bao]

aus*.mozilla.org

Comment 21

[:Atoll]

(In reply to Yuhong Bao from comment #20)
> aus*.mozilla.org

To confirm, we need to reissue SHA-1 SSL certificates for all of the following hostnames (copy-pasted from our DNS servers):

FQDN		Target	
aus.mozilla.org	CNAME	static-non-ssl.zlb.phx.mozilla.net	Edit
aus2-community.mozilla.org	CNAME	ausstage1.community.scl3.mozilla.com	Edit
aus2-dev.mozilla.org	CNAME	do-stage01.mozilla.org	Edit
aus2-static.mozilla.org	CNAME	static.zlb.phx.mozilla.net	Edit
aus2.mozilla.org	CNAME	aus01.zlb.phx.mozilla.net	Edit
aus3.mozilla.org	CNAME	aus03.zlb.phx.mozilla.net	Edit
aus4-admin.mozilla.org	CNAME	aus4-admin-prod-zlb.webapp.phx1.mozilla.com	Edit
aus4.mozilla.org	CNAME	aus4.vips.phx1.mozilla.com	Edit

aus2-staging.mozilla.org	A	63.245.209.62	Edit
aus2-test.mozilla.org	A	63.245.209.49	Edit
aus4-admin-dev.allizom.org	A	63.245.217.120	Edit


This seems like an excessively-long list. I think 'aus*.mozilla.org' means one thing to AUS developers/admins and another thing to me - since I don't have background here, I can only hope we don't have certificates for *all* of these.

(I probably missed one in my by-hand copy-paste, so don't depend on this being a comprehensive list. And I excluded mozillamessaging because Callek has asked me to delay any SHA2 reissues for those hostnames.)


So, of the above hostnames, which actually need to be converted from SHA-1 to SHA-2? Do any *.allizom hostnames need to be converted from SHA-1 to SHA-2?

Comment 22

[Yuhong Bao]

(In reply to Ben Hearsum [:bhearsum] from comment #18)
> (In reply to Richard Soderberg [:atoll] from comment #15)
> > Ben, which registrars do we need to provide SHA-2 intermediates for, to help
> > move this forward?
> 
> Doesn't matter where we get the certificates from as far as I'm concerned.
> We just need to update the pinning in
> https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.
> js#L152
> 
> If it's possible to get one from an intermediary with the same details as
> the existing DigiCert one, this is even easier.
> https://www.digicert.com/digicert-root-certificates.htm suggests that SHA2
> certs come off of a different intermediary, but I'd be happy to be wrong
> about that.

I'd suggest just issuing SHA2 certs from the SHA1 intermediate, which is allowed in the Baseline Requirements.

Comment 23

[[e-mail] (:bhearsum)]

Unfortunately, we cannot change the certs on aus, aus2, or aus3. If we were to change them to something signed by a different intermediary we would break the pinning in Firefox, and updates wouldn't work.

There's two things we could do here:
1) Update the aus4.mozilla.org cert with a sha-2 one. This can only be done if we can get a sha-2 cert signed by the same intermediary that we already pin (specified at https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.js#L154 - "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"). (If we change the cert to a sha-2 one from a different intermediary, updates won't work because cert pinning will fail.) 

2) Get a new cert for aus5.mozilla.org, update our pinning, and switch our applications to that. XP SP2 updates will continue to work for any users that updated to a version of Firefox with the new domain/pinned certs prior to sha-1 deprecation. In theory, we might be able to adjust existing users to the new domain/pinned certs with a hotfix even after that date (assuming the hotfix server is sha-2!), but this has never been tested.

Comment 24

[Yuhong Bao]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> Unfortunately, we cannot change the certs on aus, aus2, or aus3. If we were
> to change them to something signed by a different intermediary we would
> break the pinning in Firefox, and updates wouldn't work.
> 
> There's two things we could do here:
> 1) Update the aus4.mozilla.org cert with a sha-2 one. This can only be done
> if we can get a sha-2 cert signed by the same intermediary that we already
> pin (specified at
> https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.
> js#L154 - "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"). (If we change
> the cert to a sha-2 one from a different intermediary, updates won't work
> because cert pinning will fail.)

Comment 25

[Yuhong Bao]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> Unfortunately, we cannot change the certs on aus, aus2, or aus3. If we were
> to change them to something signed by a different intermediary we would
> break the pinning in Firefox, and updates wouldn't work.
> 
> There's two things we could do here:
> 1) Update the aus4.mozilla.org cert with a sha-2 one. This can only be done
> if we can get a sha-2 cert signed by the same intermediary that we already
> pin (specified at
> https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.
> js#L154 - "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"). (If we change
> the cert to a sha-2 one from a different intermediary, updates won't work
> because cert pinning will fail.) 
Is Symantec or DigiCert willing to do this?

Comment 26

[[e-mail] (:bhearsum)]

I was just speaking to Rob Strong and he told me that he's pretty sure that SSL verification of the update server bypasses Windows security, which means we don't have to worry about aus4.mozilla.org+sha1+XP SP2 after the cut-off date -- it will continue to work. Is there some way we can disable sha-1 support on an existing Windows install to verify that? If that's the case, we should just go with my option #2.


(In reply to Yuhong Bao from comment #25)
> (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > Unfortunately, we cannot change the certs on aus, aus2, or aus3. If we were
> > to change them to something signed by a different intermediary we would
> > break the pinning in Firefox, and updates wouldn't work.
> > 
> > There's two things we could do here:
> > 1) Update the aus4.mozilla.org cert with a sha-2 one. This can only be done
> > if we can get a sha-2 cert signed by the same intermediary that we already
> > pin (specified at
> > https://github.com/mozilla/gecko-dev/blob/master/browser/app/profile/firefox.
> > js#L154 - "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"). (If we change
> > the cert to a sha-2 one from a different intermediary, updates won't work
> > because cert pinning will fail.) 
> Is Symantec or DigiCert willing to do this?

I'm not sure. I don't want to waste time looking into that unless the above idea can't be proved, though.

Comment 27

[Yuhong Bao]

(In reply to Ben Hearsum [:bhearsum] from comment #26)
> I was just speaking to Rob Strong and he told me that he's pretty sure that
> SSL verification of the update server bypasses Windows security, which means
> we don't have to worry about aus4.mozilla.org+sha1+XP SP2 after the cut-off
> date -- it will continue to work. Is there some way we can disable sha-1
> support on an existing Windows install to verify that? If that's the case,
> we should just go with my option #2.
Yea, the XP SP2 problem is only for the initial browser download using IE.

Comment 28

[Robert Strong (they/them - no direct email)]

Note that the custom cert pinning is no longer used on Windows and Mac and it will no longer be used on Linux as soon as bug 1158870 is fixed. Then we just need to double check that b2g doesn't use it which I think it doesn't.

Of course this doesn't help with media.gmp-manager.certs.*

Comment 29

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
>...
> 2) Get a new cert for aus5.mozilla.org, update our pinning, and switch our
> applications to that. XP SP2 updates will continue to work for any users
> that updated to a version of Firefox with the new domain/pinned certs prior
> to sha-1 deprecation.
Note that we used essentially this process when we first added the custom cert pinning way back when and when the certs were changed recently. This is by far the less error prone path forward that I have been able to come up with and since we've used this process before we hve experience with it that shows that the issues are negligible.

Comment 30

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #29)
> (In reply to Ben Hearsum [:bhearsum] from comment #23)
> >...
> > 2) Get a new cert for aus5.mozilla.org, update our pinning, and switch our
> > applications to that. XP SP2 updates will continue to work for any users
> > that updated to a version of Firefox with the new domain/pinned certs prior
> > to sha-1 deprecation.
> Note that we used essentially this process when we first added the custom
> cert pinning way back when and when the certs were changed recently. This is
> by far the less error prone path forward that I have been able to come up
> with and since we've used this process before we hve experience with it that
> shows that the issues are negligible.

Yeah, given all this new information I agree that we should just go with this option.

atoll, can we talk specifics when you get back? I'm hoping to get the new fingerprints added to Firefox in 40.0 (ships 6 weeks from now).

Comment 31

[Yuhong Bao]

(In reply to Ben Hearsum [:bhearsum] from comment #30)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #29)
> > (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > >...
> > > 2) Get a new cert for aus5.mozilla.org, update our pinning, and switch our
> > > applications to that. XP SP2 updates will continue to work for any users
> > > that updated to a version of Firefox with the new domain/pinned certs prior
> > > to sha-1 deprecation.
> > Note that we used essentially this process when we first added the custom
> > cert pinning way back when and when the certs were changed recently. This is
> > by far the less error prone path forward that I have been able to come up
> > with and since we've used this process before we hve experience with it that
> > shows that the issues are negligible.
> 
> Yeah, given all this new information I agree that we should just go with
> this option.
> 
> atoll, can we talk specifics when you get back? I'm hoping to get the new
> fingerprints added to Firefox in 40.0 (ships 6 weeks from now).

AFAIK aus3's Thawte cert expires in Sept 2017, and aus4's DigiCert cert expires in 2016.

Comment 32

[[e-mail] (:bhearsum)]

(In reply to Yuhong Bao from comment #31)
> (In reply to Ben Hearsum [:bhearsum] from comment #30)
> > (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> > comment #29)
> > > (In reply to Ben Hearsum [:bhearsum] from comment #23)
> > > >...
> > > > 2) Get a new cert for aus5.mozilla.org, update our pinning, and switch our
> > > > applications to that. XP SP2 updates will continue to work for any users
> > > > that updated to a version of Firefox with the new domain/pinned certs prior
> > > > to sha-1 deprecation.
> > > Note that we used essentially this process when we first added the custom
> > > cert pinning way back when and when the certs were changed recently. This is
> > > by far the less error prone path forward that I have been able to come up
> > > with and since we've used this process before we hve experience with it that
> > > shows that the issues are negligible.
> > 
> > Yeah, given all this new information I agree that we should just go with
> > this option.
> > 
> > atoll, can we talk specifics when you get back? I'm hoping to get the new
> > fingerprints added to Firefox in 40.0 (ships 6 weeks from now).
> 
> AFAIK aus3's Thawte cert expires in Sept 2017, and aus4's DigiCert cert
> expires in 2016.

We'll address these (if we feel it's worthwhile) closer to their expiry dates. If we switch to aus5 in the next month, that means we'll have been off of aus4 for over a year before its cert expires, and off of aus3 for many years. This is explicitly not in scope for this bug, we're only dealing with making sure _current_ versions of Firefox update from a server with a sha-2 cert.

Comment 33

[:Atoll]

You're welcome to chat with me - but, if you wish, you can have anyone in Webops simply issue an EV (or not) aus5 cert for you, and that'll include the new intermediate for your purposes. I'll be back next week if you'd rather wait to issue that cert. I don't have any specific knowledge about which intermediate it would be issued with - and once anyone issues it, I'll share their knowledge :)

Comment 34

[[e-mail] (:bhearsum)]

Filed bug 1179339 to get aus5.m.o set-up. We'll deal with update in-tree pinning and other non-IT stuff here.

Comment 35

[[e-mail] (:bhearsum)]

Robert, I got reminded today that we don't do SSL verifications now that we have MAR signing on all platforms. Am I right that this means that we don't need to update in-tree fingerprints in app.update.certs.*? If so, what's the earliest Gecko version that applies to?

I also remembered that we should probably buy a new SHA-1 cert for aus4.mozilla.org, to keep it valid for as long as possible...

Comment 36

[Robert Strong (they/them - no direct email)]

Linux mar signing landed just a short while ago for Firefox 42 and the cert check will be disabled for Firefox 42 in bug 1151485. Earlier versions will still use the aus4 and the sha1 cert.

Comment 37

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #36)
> Linux mar signing landed just a short while ago for Firefox 42 and the cert
> check will be disabled for Firefox 42 in bug 1151485. Earlier versions will
> still use the aus4 and the sha1 cert.

Thanks! In that case, the only thing I'll do in this bug is s/aus4.mozilla.org/aus5.mozilla.org/g, and I won't even touch the fingerprints. That will land only on mozilla-central, which will make its way all the way to mozilla-release in November.

I'll let ESR38 continue to use aus4.mozilla.org, which looks like it will continue to exist until June, 2016. The next ESR (45) will pick up aus5.mozilla.org, of course.

Comment 38

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #37)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #36)
> > Linux mar signing landed just a short while ago for Firefox 42 and the cert
> > check will be disabled for Firefox 42 in bug 1151485. Earlier versions will
> > still use the aus4 and the sha1 cert.
> 
> Thanks! In that case, the only thing I'll do in this bug is
> s/aus4.mozilla.org/aus5.mozilla.org/g, and I won't even touch the
> fingerprints. That will land only on mozilla-central, which will make its
> way all the way to mozilla-release in November.

Looks like we need to update release configs when this uplifts to Beta. http://mxr.mozilla.org/build/search?string=aus4 isn't showing anything else that needs updating in a timely manner, though we should update the defaults to final-verification.sh and other scripts once aus5 is the default everywhere.

Comment 39

[[e-mail] (:bhearsum)]

Attachment: update gecko repo update server references

Robert, I'm not sure if all of these are in your wheelhouse so feel free to redirect me if I need different review for eg, the security pinning parts.

This was pretty straightforward with the exception of still needing to pin for GMP updates. I think I did it correctly by leaving aus4.m.o in there with the old issuerName...

Comment 40

[[e-mail] (:bhearsum)]

Attachment: move thunderbird to aus5

This is pretty much the same as Firefox. I don't think we have MAR signing for Thunderbird, so the cert pinning is still necessary.

Comment 41

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #40)
> Created attachment 8641700 [details] [diff] [review]
> move thunderbird to aus5
> 
> This is pretty much the same as Firefox. I don't think we have MAR signing
> for Thunderbird, so the cert pinning is still necessary.
I believe they have it for Windows since the maintenance service requires it though they likely haven't enabled it for Mac and Linux.

Comment 42

[Robert Strong (they/them - no direct email)]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

Please give dhylands a heads up about the change to b2g.

Android doesn't use the app update code and has a custom implementation. Please have someone on the android team review their change just in case.

Please have someone else review StaticHPKPins.h

Note: as I understand it the entries in StaticHPKPins.h are for telemetry reporting and they aren't actual cert pins since it was decided awhile back that aus will not be pinned for various reasons. See bug 1063111

diff --git a/modules/libpref/init/all.js b/modules/libpref/init/all.js
--- a/modules/libpref/init/all.js
+++ b/modules/libpref/init/all.js
@@ -4952,7 +4952,7 @@ pref("browser.search.official", true);
 //pref("media.gmp-manager.url.override", "");
 
 // Update service URL for GMP install/updates:
-pref("media.gmp-manager.url", "https://aus4.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml");
+pref("media.gmp-manager.url", "https://aus5.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml");
 
 // When |media.gmp-manager.cert.requireBuiltIn| is true or not specified the
 // final certificate and all certificates the connection is redirected to before
@@ -4979,8 +4979,8 @@ pref("media.gmp-manager.cert.requireBuil
 pref("media.gmp-manager.cert.checkAttributes", true);
 pref("media.gmp-manager.certs.1.issuerName", "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US");
 pref("media.gmp-manager.certs.1.commonName", "aus4.mozilla.org");
-pref("media.gmp-manager.certs.2.issuerName", "CN=Thawte SSL CA,O=\"Thawte, Inc.\",C=US");
-pref("media.gmp-manager.certs.2.commonName", "aus4.mozilla.org");
+pref("media.gmp-manager.certs.2.issuerName", "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US");
+pref("media.gmp-manager.certs.2.commonName", "aus5.mozilla.org");
 #endif

This doesn't make sense to me since media.gmp-manager.certs.1. will always fail. These entries should be for the cert that is used which should be media.gmp-manager.certs.1. and the backup cert which should be media.gmp-manager.certs.2.

Everything else looks good except for that.

Comment 43

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #42)
> diff --git a/modules/libpref/init/all.js b/modules/libpref/init/all.js
> --- a/modules/libpref/init/all.js
> +++ b/modules/libpref/init/all.js
> @@ -4952,7 +4952,7 @@ pref("browser.search.official", true);
>  //pref("media.gmp-manager.url.override", "");
>  
>  // Update service URL for GMP install/updates:
> -pref("media.gmp-manager.url",
> "https://aus4.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/
> %LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.
> xml");
> +pref("media.gmp-manager.url",
> "https://aus5.mozilla.org/update/3/GMP/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/
> %LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.
> xml");
>  
>  // When |media.gmp-manager.cert.requireBuiltIn| is true or not specified the
>  // final certificate and all certificates the connection is redirected to
> before
> @@ -4979,8 +4979,8 @@ pref("media.gmp-manager.cert.requireBuil
>  pref("media.gmp-manager.cert.checkAttributes", true);
>  pref("media.gmp-manager.certs.1.issuerName", "CN=DigiCert Secure Server
> CA,O=DigiCert Inc,C=US");
>  pref("media.gmp-manager.certs.1.commonName", "aus4.mozilla.org");
> -pref("media.gmp-manager.certs.2.issuerName", "CN=Thawte SSL CA,O=\"Thawte,
> Inc.\",C=US");
> -pref("media.gmp-manager.certs.2.commonName", "aus4.mozilla.org");
> +pref("media.gmp-manager.certs.2.issuerName", "CN=DigiCert SHA2 Secure
> Server CA,O=DigiCert Inc,C=US");
> +pref("media.gmp-manager.certs.2.commonName", "aus5.mozilla.org");
>  #endif
> 
> This doesn't make sense to me since media.gmp-manager.certs.1. will always
> fail. These entries should be for the cert that is used which should be
> media.gmp-manager.certs.1. and the backup cert which should be
> media.gmp-manager.certs.2.
> 
> Everything else looks good except for that.

Per IRC, we also need to get a backup cert for aus5 - I've re-opened bug 1179339 for that. I'll update this bug with a new patch with the new fingerprints when that's done, but I'd like to get review for the other aspects in the meantime - there's not much time left before 42 hits Aurora.

Comment 44

[[e-mail] (:bhearsum)]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

Dave, Mark, Monica - adding you as reviewers for the b2g, mobile, and security parts of this respectively. The context here is that we're switching the update server from aus4.mozilla.org to aus5.mozilla.org.

Comment 45

[Dave Hylands [:dhylands]]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

I can rubber stamp it. I mean this is fine as long as there is a server running on aus5 and its serving up the same stuff.

Comment 46

[[e-mail] (:bhearsum)]

(In reply to Dave Hylands [:dhylands] from comment #45)
> Comment on attachment 8641699 [details] [diff] [review]
> update gecko repo update server references
> 
> I can rubber stamp it. I mean this is fine as long as there is a server
> running on aus5 and its serving up the same stuff.

Yep, this is the case! aus5 is just a domain with a new SSL certificate - the backends are the same ones as aus4.

Comment 47

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

Review of attachment 8641699 [details] [diff] [review]:
-----------------------------------------------------------------

r=me for the security/manager/... changes with comment addressed.

::: security/manager/ssl/StaticHPKPins.h
@@ +751,4 @@
>    { "apps.facebook.com", true, false, false, -1, &kPinset_facebook },
>    { "appspot.com", true, false, false, -1, &kPinset_google_root_pems },
>    { "aus4.mozilla.org", true, true, true, 3, &kPinset_mozilla },
> +  { "aus5.mozilla.org", true, true, true, 3, &kPinset_mozilla },

This file needs to be generated by running security/manager/tools/genHPKPStaticPins.js. The magic invocation (for me on a linux box in the root of the source tree after building) is "./obj-x86_64-unknown-linux-gnu/dist/bin/run-mozilla.sh ./obj-x86_64-unknown-linux-gnu/dist/bin/xpcshell security/manager/tools/genHPKPStaticPins.js `pwd`/security/manager/tools/PreloadedHPKPins.json `pwd`/security/manager/ssl/tests/unit/tlsserver/default-ee.der `pwd`/security/manager/ssl/StaticHPKPins.h"

Alternatively, just wait for the automatic updater to regenerate it for you on the Saturday after this gets checked in (be careful about uplifts, though).

Comment 48

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

I assume the mobile GMP downloads will be fine, since it's using the same code as desktop.

Mobile uses a Java updater, which should be fine with these changes, but I am looping Snorp in just to be sure.

Comment 49

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8641699 [details] [diff] [review]
update gecko repo update server references

Review of attachment 8641699 [details] [diff] [review]:
-----------------------------------------------------------------

Seems fine. We don't use pinning in the java updater (sigh), and sha2 should be fine.

Comment 50

[Joshua Cranmer [:jcranmer]]

Comment on attachment 8641700 [details] [diff] [review]
move thunderbird to aus5

Review of attachment 8641700 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me. I'm not well-informed when it comes to the TB update infrastructure, so I'm trusting your judgement that this is necessary and correct.

Comment 51

[[e-mail] (:bhearsum)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #47)
> Comment on attachment 8641699 [details] [diff] [review]
> update gecko repo update server references
> 
> Review of attachment 8641699 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r=me for the security/manager/... changes with comment addressed.
> 
> ::: security/manager/ssl/StaticHPKPins.h
> @@ +751,4 @@
> >    { "apps.facebook.com", true, false, false, -1, &kPinset_facebook },
> >    { "appspot.com", true, false, false, -1, &kPinset_google_root_pems },
> >    { "aus4.mozilla.org", true, true, true, 3, &kPinset_mozilla },
> > +  { "aus5.mozilla.org", true, true, true, 3, &kPinset_mozilla },
> 
> This file needs to be generated by running
> security/manager/tools/genHPKPStaticPins.js. The magic invocation (for me on
> a linux box in the root of the source tree after building) is
> "./obj-x86_64-unknown-linux-gnu/dist/bin/run-mozilla.sh
> ./obj-x86_64-unknown-linux-gnu/dist/bin/xpcshell
> security/manager/tools/genHPKPStaticPins.js
> `pwd`/security/manager/tools/PreloadedHPKPins.json
> `pwd`/security/manager/ssl/tests/unit/tlsserver/default-ee.der
> `pwd`/security/manager/ssl/StaticHPKPins.h"
> 
> Alternatively, just wait for the automatic updater to regenerate it for you
> on the Saturday after this gets checked in (be careful about uplifts,
> though).

Seems like using the automatic updater should be fine...it's not going to be an issue if it doesn't get updated for a day or two after the patch lands AFAIK. Which part of uplifts do I need to be careful of? I'll probably uplift to whereever 42 is when this lands (hopefully just Aurora), and let it ride from there.

Comment 52

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Ok - if you use the automatic updater, just don't include the changes to StaticHPKPins.h. Uplifting to Aurora should be no problem (again, just don't include changes to StaticHPKPins.h).

Comment 53

[[e-mail] (:bhearsum)]

Attachment: updated patch with proper pinning for gmp

Updated patch with proper pinning now that we've got a backup cert: https://fox2mike.pastebin.mozilla.org/8843112

I'll need to update the comm patch for this as well.

Comment 54

[[e-mail] (:bhearsum)]

Attachment: update backup certs for comm

Comment 55

[Robert Strong (they/them - no direct email)]

Comment on attachment 8649518 [details] [diff] [review]
updated patch with proper pinning for gmp

Note: I just reviewed the media.gmp-manager.certs.* preferences for the cert checks and didn't verify that they match the certificates you acquired.

Comment 56

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0d376102df5f

Comment 57

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0d376102df5f

Comment 58

[[e-mail] (:bhearsum)]

I verified this on Linux and Windows today.

Comment 59

[[e-mail] (:bhearsum)]

Comment on attachment 8649518 [details] [diff] [review]
updated patch with proper pinning for gmp

This patch needs to be uplifte as far as Gecko 42 (currently in Aurora) to make sure that we switch to an update server with a SHA-2 cert before SHA-1 is deprecated on January 1st. (Technically, Gecko 43 will be released before that date, but it's bumping up close to the end of year, and I don't want to take any chances of this not making it.)

This patch is very low risk - I've already verified that updates continue to work on Nightly with it applied. There is no user facing impact or change.

Comment 60

[[e-mail] (:bhearsum)]

Turns out I bungled the pinning. The O field of the issuer has quotes around it. As landed, I can't update GMP plugins when the backup cert was enabled. I fixed the pref in my profile and it installed fine. I'll land a bustage fix for that once inbound re-opens.

Comment 61

[[e-mail] (:bhearsum)]

Attachment: patch for comm with fixed pinning

Here's an updated patch for Thunderbird with proper pinning.

Comment 62

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/90fc782afd1e

Comment 63

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/90fc782afd1e

Comment 64

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #60)
> Turns out I bungled the pinning. The O field of the issuer has quotes around
> it. As landed, I can't update GMP plugins when the backup cert was enabled.
> I fixed the pref in my profile and it installed fine. I'll land a bustage
> fix for that once inbound re-opens.

I verified that this is fixed in the latest Nightly.

Comment 65

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8649518 [details] [diff] [review]
updated patch with proper pinning for gmp

Bye bye sha-1!

Comment 66

[[e-mail] (:bhearsum)]

(In reply to David Keeler (on PTO) [:keeler] (use needinfo?) from comment #52)
> Ok - if you use the automatic updater, just don't include the changes to
> StaticHPKPins.h. Uplifting to Aurora should be no problem (again, just don't
> include changes to StaticHPKPins.h).

David, I still don't see aus5.mozilla.org in https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h - should I be concerned about that?

Comment 67

[[e-mail] (:bhearsum)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/3a38dba3692d

Comment 68

[Ryan VanderMeulen [:RyanVM]]

Please don't forget to set the status flag when landing uplifts.

Comment 69

[:Cykesiopka]

(In reply to Ben Hearsum (:bhearsum) from comment #66)
> David, I still don't see aus5.mozilla.org in
> https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/
> StaticHPKPins.h - should I be concerned about that?

This turns out to be due to Bug 1197607.

Comment 70

[[e-mail] (:bhearsum)]

(In reply to Cykesiopka from comment #69)
> (In reply to Ben Hearsum (:bhearsum) from comment #66)
> > David, I still don't see aus5.mozilla.org in
> > https://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/
> > StaticHPKPins.h - should I be concerned about that?
> 
> This turns out to be due to Bug 1197607.

Ah, thank you!

Comment 71

[Nick Thomas [:nthomas] (UTC+12)]

mozilla-central has aus5 in the pinning manifest now - https://hg.mozilla.org/mozilla-central/rev/bb0711d3f60a#l1.62. I'm going to get this uplifted to aurora and esr38 for sure, but does it need to go anywhere else ?

Comment 72

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #71)
> mozilla-central has aus5 in the pinning manifest now -
> https://hg.mozilla.org/mozilla-central/rev/bb0711d3f60a#l1.62. I'm going to
> get this uplifted to aurora and esr38 for sure, but does it need to go
> anywhere else ?

I think those are the only places...I totally forgot about getting the original patch on esr38 :S

Comment 73

[[e-mail] (:bhearsum)]

Comment on attachment 8649518 [details] [diff] [review]
updated patch with proper pinning for gmp

This needs to be backported to ESR38 for the same reason it had to be backedported to Aurora - we need to use an update server with a SHA-2 cert before EOY.

Comment 74

[Robert Strong (they/them - no direct email)]

Comment on attachment 8649518 [details] [diff] [review]
updated patch with proper pinning for gmp

esr38 does not have bug 1158870 so you will need to fix the update server attributes for it with a new patch.

Comment 75

[Robert Strong (they/them - no direct email)]

Also, is the deprecation of sha1 going to be backported to esr38? I haven't heard one way or the other as to if it would be but it would kind of surprise me.

Comment 76

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #74)
> Comment on attachment 8649518 [details] [diff] [review]
> updated patch with proper pinning for gmp
> 
> esr38 does not have bug 1158870 so you will need to fix the update server
> attributes for it with a new patch.

Thanks for catching that. I'll put together a new patch for it.

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #75)
> Also, is the deprecation of sha1 going to be backported to esr38? I haven't
> heard one way or the other as to if it would be but it would kind of
> surprise me.

I'm not sure, but it's easy enough to backport, and I'd rather by consistent...

Comment 77

[[e-mail] (:bhearsum)]

Attachment: switch esr38 to aus5 w/ updated pinning

So to summarize, the following needs to be done still:
* Land this patch on esr38
* Make sure the static pins get updated on aurora + esr38.
** This seems to be broken on Aurora right now, eg: http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-aurora%20periodic%20file%20update/builds/0/steps/run_script/logs/stdio
** The builder doesn't exist at all on esr38
** I'm a bit wary of doing it by hand because of this part of the diff:
   1.119 -static const PRTime kPreloadPKPinsExpirationTime = INT64_C(1446891927723000);
   1.120 +static const PRTime kPreloadPKPinsExpirationTime = INT64_C(1449780589679000);

Comment 78

[Robert Strong (they/them - no direct email)]

Comment on attachment 8657192 [details] [diff] [review]
switch esr38 to aus5 w/ updated pinning

Client versions prior to this change which include release 38 will still be using sha1 and aus4 and I would rather be consistent with that so please verify. Thanks

Comment 79

[Robert Strong (they/them - no direct email)]

Also, I don't like making unnecessary changes on esr at all so if this is not necessary I definitely don't want this change.

Comment 80

[Joshua Cranmer [:jcranmer]]

Comment on attachment 8652866 [details] [diff] [review]
patch for comm with fixed pinning

Review of attachment 8652866 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not entirely certain about the certs.2 field, but I did verify the certs.1 by looking at the certificate details in Firefox.

Comment 81

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #78)
> Comment on attachment 8657192 [details] [diff] [review]
> switch esr38 to aus5 w/ updated pinning
> 
> Client versions prior to this change which include release 38 will still be
> using sha1 and aus4 and I would rather be consistent with that so please
> verify. Thanks

Julien, do you know if SHA-1 is going to be deprecated on ESR38? We don't want to make unnecessary changes to that branch if we don't have to.

Comment 82

[Julien Vehent]

I don't know of any timeline to deprecate SHA-1 for signatures verification. Looping in rbarnes who will know more.

Comment 83

[[e-mail] (:bhearsum)]

Comment on attachment 8652866 [details] [diff] [review]
patch for comm with fixed pinning

Let's get this comm patch landed on Aurora, too.

Comment 84

[[e-mail] (:bhearsum)]

Status update, mostly for my own sanity:
* Firefox 42 and newer (currently Aurora and Central) have been switched to aus5. I've verified that the static pins have been updated in both of these repos.
* I've landed the aus4 -> aus5 switch for Thunderbird on comm-central, need to backport to comm-aurora still.
* Waiting for response from Richard on whether or not ESR38 will enforce SHA-1 deprecation. If it will, I need to backport to both mozilla-esr38 & comm-esr38. If it won't, nothing to do for ESR.

Comment 85

[Sylvestre Ledru [:Sylvestre]]

Ben, do you mind creating a new bug for this? Reusing a fixed bug (status flags marks it as fixed too) can cause the bug to be missed by relman or the sheriff.

Comment 86

[[e-mail] (:bhearsum)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #85)
> Ben, do you mind creating a new bug for this? Reusing a fixed bug (status
> flags marks it as fixed too) can cause the bug to be missed by relman or the
> sheriff.

Which part are you suggesting should be a new bug? There's only patch per branch for Firefox, and one patch per branch for Thunderbird. So...that's only one patch per repo.

Comment 87

[[e-mail] (:bhearsum)]

Comment on attachment 8652866 [details] [diff] [review]
patch for comm with fixed pinning

Sorry, wrong approval flag was requested. Maybe that was the source of your confusion, Sylvestre?

Comment 88

[[e-mail] (:bhearsum)]

Comment on attachment 8652866 [details] [diff] [review]
patch for comm with fixed pinning

Fallen told me the other day that I have blanked approval for RelEng stuff, so I went ahead and landed this on comm-aurora: https://hg.mozilla.org/releases/comm-aurora/rev/4b4e76877e1f

Comment 89

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Ben Hearsum (:bhearsum) from comment #81)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #78)
> > Comment on attachment 8657192 [details] [diff] [review]
> > switch esr38 to aus5 w/ updated pinning
> > 
> > Client versions prior to this change which include release 38 will still be
> > using sha1 and aus4 and I would rather be consistent with that so please
> > verify. Thanks
> 
> Julien, do you know if SHA-1 is going to be deprecated on ESR38? We don't
> want to make unnecessary changes to that branch if we don't have to.

I can fairly confidently say that any sha-1 deprecation code will not be backported to esr38.

Comment 90

[[e-mail] (:bhearsum)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #89)
> (In reply to Ben Hearsum (:bhearsum) from comment #81)
> > (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> > comment #78)
> > > Comment on attachment 8657192 [details] [diff] [review]
> > > switch esr38 to aus5 w/ updated pinning
> > > 
> > > Client versions prior to this change which include release 38 will still be
> > > using sha1 and aus4 and I would rather be consistent with that so please
> > > verify. Thanks
> > 
> > Julien, do you know if SHA-1 is going to be deprecated on ESR38? We don't
> > want to make unnecessary changes to that branch if we don't have to.
> 
> I can fairly confidently say that any sha-1 deprecation code will not be
> backported to esr38.

If this is the case, there's nothing left to do here. I'll wait for rbarnes to confirm that, since he's already needinfo'ed.

Comment 91

[Richard Barnes [:rbarnes]]

The only SHA-1 deprecation code we have was landed in Bug 942515.  AFAIK, it is not going to be back-ported to esr38.