803531 - renew authenticode certificates in september 2013

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

We should look at using a different root this time, too, one that lets us generate our own private key + CSR. If we did this, we wouldn't have to deal with IE + Windows, and could kill keymaster.

Comment 1

[[e-mail] (:bhearsum)]

We should update the add-on hotfix fingerprints again, like https://bugzilla.mozilla.org/show_bug.cgi?id=803596

Comment 2

[[e-mail] (:bhearsum)]

Expires on 10/18/2013 , we should renew mid-September so that we can do bug 874513 in good time.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #0)
> We should look at using a different root this time, too, one that lets us
> generate our own private key + CSR. If we did this, we wouldn't have to deal
> with IE + Windows, and could kill keymaster.

Note to self: talk to opsec about this first.

Comment 4

[[e-mail] (:bhearsum)]

It's likely that we can use a 3 year 2048-bit RSA for this. Another note to self: check with kang again before doing it.

Comment 5

[[e-mail] (:bhearsum)]

More notes from e-mail with dveditz and bsmith:
* Consider getting EV signing certificates (https://blogs.msdn.com/b/ie/archive/2012/08/14/microsoft-smartscreen-amp-extended-validation-ev-code-signing-certificates.aspx?Redirected=true). bsmith says "In particular, these EV code signing certificates should make the process of opening the downloading/running the installer faster, especially on Windows 8+."
* We need to make sure that the certs we get have "Netscape Object Signing" bits on them, otherwise XPI signing won't work.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #5)
> * Consider getting EV signing certificates
> (https://blogs.msdn.com/b/ie/archive/2012/08/14/microsoft-smartscreen-amp-
> extended-validation-ev-code-signing-certificates.aspx?Redirected=true).
> bsmith says "In particular, these EV code signing certificates should make
> the process of opening the downloading/running the installer faster,
> especially on Windows 8+."

I did some investigation into this and found out that these require an HSM of some sorts. With the way our automation works at this time we can't support this. We are planning to look at HSMs for all signing in the future, and will re-visit at this time.


Right now I'm looking at DigiCert for renewal. We already have a full enterprise account with them, which I now have access to, and they claim that I'll be able to submit my own CSR to get the job done. I'm working through this process right now with a bit of help from Chris Turra, who has worked with them for a bunch of SSL certificates.

> * We need to make sure that the certs we get have "Netscape Object Signing"
> bits on them, otherwise XPI signing won't work.

I'm also told that the Authenticode certificate we'll get from them will have these bits. Once I have the first certificate in hand I'll be doing some signing of Windows builds and XPIs to make sure that everything checks out before we fully deploy the new certificates.

We're currently waiting for DigiCert to do some verifications and I hope to have the first cert by sometime tomorrow.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: update in-tree fingerprint

Dave, this patch has the SHA1 fingerprint of our new certificate. I signed a test add-on with the new certificate if there's any verification you want to do: https://people.mozilla.org/~bhearsum/test-signed.xpi (I had Felipe look at a similar add-on yesterday, so I don't anticipate any issues).

bug 874513 is going to deal with updating existing installs through a hotfix.

Comment 8

[Dave Townsend [:mossop]]

Comment on attachment 807191 [details] [diff] [review]
update in-tree fingerprint

This fingerprint matches that of the cert used to sign the test add-on which was issued to "Mozilla Corporation" by "DigiCert Assured ID Code Signing CA-1" expiring "Sep 21 12:00:00 2016 GMT".

Assuming that all matches what you expect then this looks good.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Dave Townsend (:Mossop) from comment #8)
> Comment on attachment 807191 [details] [diff] [review]
> update in-tree fingerprint
> 
> This fingerprint matches that of the cert used to sign the test add-on which
> was issued to "Mozilla Corporation" by "DigiCert Assured ID Code Signing
> CA-1" expiring "Sep 21 12:00:00 2016 GMT".

Looks right to me. Thanks for looking at this so quickly.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 807191 [details] [diff] [review]
update in-tree fingerprint

Pushed to inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/951a95bca7c2

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Ben, I think we'll need to update
  http://hg.mozilla.org/mozilla-central/file/default/browser/branding/official/branding.nsi#l24
and other brandings, on appropriate trees. 

Also figure out how to deploy to avoid mismatched stubs/installers and what they're signed with.

Comment 12

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/951a95bca7c2

Comment 13

[[e-mail] (:bhearsum)]

Re-opening because there's still more to do here.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #11)
> Ben, I think we'll need to update
>  
> http://hg.mozilla.org/mozilla-central/file/default/browser/branding/official/
> branding.nsi#l24
> and other brandings, on appropriate trees. 
> 

Yikes, thanks for catching that Nick. Looks like the complete list of files is:
https://mxr.mozilla.org/mozilla-central/source/browser/branding/aurora/branding.nsi
https://mxr.mozilla.org/mozilla-central/source/browser/branding/nightly/branding.nsi
https://mxr.mozilla.org/mozilla-central/source/browser/branding/official/branding.nsi
https://mxr.mozilla.org/mozilla-central/source/browser/branding/unofficial/branding.nsi
https://mxr.mozilla.org/mozilla-central/source/browser/installer/windows/nsis/maintenanceservice_installer.nsi
https://mxr.mozilla.org/mozilla-central/source/browser/installer/windows/nsis/defines.nsi.in
https://mxr.mozilla.org/mozilla-central/source/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi

I don't see any reason why we shouldn't be updating all of them on all of the trees, since we'll be signing with the new certificate everywhere.

> Also figure out how to deploy to avoid mismatched stubs/installers and what
> they're signed with.

I think this shouldn't be an issue. We don't need to worry about Release and Beta at all, because they always point at specific files that don't get overwritten.

For Aurora, we point at both the full and stub installer in latest-mozilla-aurora, which will get updated almost at the same instant. If we're worried about this, we could repoint these entries at a dated directory until we're sure that both the full and stub installer are up-to-date. IIRC, nightly/ isn't on the CDN so we don't need to worry about flushing that cache.

For Nightly I'm not sure what the state of things is. We have a firefox-nightly-latest which points at the full installer latest-mozilla-central, but no nightly stub entry. That makes me think that this is unused. In any case, I think whatever we do for Aurora should apply here.

Does the above sound sensible to you, Nick?

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #14)
> For Nightly I'm not sure what the state of things is. We have a
> firefox-nightly-latest which points at the full installer
> latest-mozilla-central, but no nightly stub entry.

Relman, any idea about this?

Comment 16

[[e-mail] (:bhearsum)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/cfea8a18f177
https://hg.mozilla.org/releases/mozilla-beta/rev/13d8be2c624f
https://hg.mozilla.org/releases/mozilla-release/rev/bd0274331077
https://hg.mozilla.org/releases/mozilla-esr17/rev/549854269e42
https://hg.mozilla.org/releases/mozilla-esr24/rev/fe3604b05f97

Left to do here:
* Update in-tree stub installer cert details.
* Finalize plan for rolling out new certs to signing servers (see comments 11, 14 and 15), and do it.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: update stub installer issuer

Here's the relevant certificate info:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            05:11:ea:f8:57:9e:26:62:be:62:2d:e5:ae:0c:d4:08
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Assured ID Code Signing CA-1
        Validity
            Not Before: Sep 17 00:00:00 2013 GMT
            Not After : Sep 21 12:00:00 2016 GMT
        Subject: C=US, ST=CA, L=Mountain View, O=Mozilla Corporation, CN=Mozilla Corporation


AFAICT, the code looks at the CN in the subject and issuer.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 807835 [details] [diff] [review]
update stub installer issuer

Don't think I have sufficient juice to r+ this, so just some comments.

browser/branding/unofficial/branding.nsi seems to be missing.

>diff --git a/browser/installer/windows/nsis/maintenanceservice_installer.nsi b/browser/installer/windows/nsis/maintenanceservice_installer.nsi
>-  ; WriteRegStr HKLM "${FallbackKey}\0" "issuer" "Thawte Code Signing CA - G2"
>+  ; WriteRegStr HKLM "${FallbackKey}\0" "issuer" "DigiCert Assured ID Code Signing CA-1"
>diff --git a/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi b/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi
>@@ -202,7 +202,7 @@ Section "MaintenanceService"
>-  WriteRegStr HKLM "${FallbackKey}\0" "issuer" "Thawte Code Signing CA - G2"
>+  WriteRegStr HKLM "${FallbackKey}\0" "issuer" "DigiCert Assured ID Code Signing CA-1"

There is a comment slightly outside the context which indicate this is for testing:
  ; Included here for debug purposes only.
  ; These keys are used to bypass the installation dir is a valid installation
  ; check from the service so that tests can be run.

Do we need to resign some in-tree binaries using the new cert ?

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum [:bhearsum] from comment #14)
> Does the above sound sensible to you, Nick?

I think that simultaneous deployment of stub and full installer will cover it for new builds. So a no-op except for maybe Aurora with CDN.

Across all branches we can't do anything about older stub installers failing to accept a full installer with a newer cert. The user will get an error and get prompted to go back to mozilla.com (or whatever) to download the full installer (again!). Not great but hopefully uncommon.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #18)
> >diff --git a/browser/installer/windows/nsis/maintenanceservice_installer.nsi b/browser/installer/windows/nsis/maintenanceservice_installer.nsi
> >-  ; WriteRegStr HKLM "${FallbackKey}\0" "issuer" "Thawte Code Signing CA - G2"
> >+  ; WriteRegStr HKLM "${FallbackKey}\0" "issuer" "DigiCert Assured ID Code Signing CA-1"
> >diff --git a/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi b/toolkit/components/maintenanceservice/bootstrapinstaller/maintenanceservice_installer.nsi
> >@@ -202,7 +202,7 @@ Section "MaintenanceService"
> >-  WriteRegStr HKLM "${FallbackKey}\0" "issuer" "Thawte Code Signing CA - G2"
> >+  WriteRegStr HKLM "${FallbackKey}\0" "issuer" "DigiCert Assured ID Code Signing CA-1"
> 
> There is a comment slightly outside the context which indicate this is for
> testing:
>   ; Included here for debug purposes only.
>   ; These keys are used to bypass the installation dir is a valid
> installation
>   ; check from the service so that tests can be run.
> 
> Do we need to resign some in-tree binaries using the new cert ?

Rob or Brian, any insight into this?

Comment 21

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #19)
> (In reply to Ben Hearsum [:bhearsum] from comment #14)
> > Does the above sound sensible to you, Nick?
> 
> I think that simultaneous deployment of stub and full installer will cover
> it for new builds. So a no-op except for maybe Aurora with CDN.

Huh, I didn't know that Aurora was on the CDN. We should probably flush the CDN cache for its stub and full installer right after the first nightlies with the new certs happen.

> Across all branches we can't do anything about older stub installers failing
> to accept a full installer with a newer cert. The user will get an error and
> get prompted to go back to mozilla.com (or whatever) to download the full
> installer (again!). Not great but hopefully uncommon.

We're talking about people that previously downloaded a stub installer, and then waited a day or more to run it, right?

Comment 22

[Brian R. Bondy [:bbondy]]

If you are changing the issuer and name btw then be very careful for how this works for updates
http://dxr.mozilla.org/mozilla-central/source/browser/installer/windows/nsis/shared.nsh#l700

I think that the old binaries should be signed with the old authenticode cert, and then every binary after that should be signed with the new certs.

Comment 23

[Brian R. Bondy [:bbondy]]

Sorry I was still writing and I accidentally submitted the message.
I just meant that if you are changing the name and issuer then please test it out before deploying it.  The users will have the old binaries on their machine with the old cert, and you should be able to update both the registry keys in shared.nsh and the binaries at once to update to the new version.  They must both be updated at the same time, but please test an update for this, and 1 update after such a change.  

The fallback key is installed on each of the test servers, it should stay the same if possible for now.

Comment 24

[Brian R. Bondy [:bbondy]]

So here's the process as I see it:
- User has binaries on their machine signed with Cert1 and has registry key info matching Cert1.
- User downloads a new update of a MAR, MAR signature didn't change, no problem there.
- User sees update and checks the signature of its current updater.exe which is signed with Cert1 with the registry info of Cert1.   Everything matches so it's good to proceed with the update through the maintenance service.
- User applies the update, at this point they have binaries signed with the new Cert2 but registry of Cert1.
- User has PostUpdate run for them which updates the registry info in HKLM to Cert2. 

I recommend testing:
1) An update from Cert1 to Cert2 to verify that you can upgrade properly.
2) One update after that, an update from Cert2 to Cert2 to verify the Cert2 info is in the registry correctly and will pass checks.
3) Test both from updated paths and freshly installed paths from both the full installer and stub installer.

Note that Authenticode signatures are only checked by the maintenance service, so in the worst case user's would restart to update, but it wouldn't update if a signature failed.  Then the next time they restarted after that they would fallback with a UAC prompt which doesn't check the authenticode signature, and that would pass even if there was a mismatch.  

The safest thing to do is to keep the issuer and name the same, but if that isn't possible just make sure you test as per the above.

The MAR signatures must not change, for the fallback key stuff I'd leave this the same if possible or we may have to reconfigure some stuff on each build server.  Test a DEP build after these changes though to make sure nothing got broken. If you have a problem with the fallback key some of the tests will definitely fail.

Comment 25

[Brian R. Bondy [:bbondy]]

If an update is consumed by a limited user account who has their install installed into a low integrity location (just their appdir which they have write access to), then I think the /PostUpdate doesn't run elevated so the HKLM keys won't be updated.  But in that situation the maintenance service isn't used to update anyway. And the maintenance service is the only thing that uses those keys.   So nothing to worry too much about here, but it is an extra test case to try.

Comment 26

[[e-mail] (:bhearsum)]

Thanks for all the detailed information Brian. I'm doing a build and setting up test updates as described in your comment #24. I'll post details once it's been set-up.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum [:bhearsum] from comment #21)
> Huh, I didn't know that Aurora was on the CDN. We should probably flush the
> CDN cache for its stub and full installer right after the first nightlies
> with the new certs happen.

I went back to check this and discovered I was incorrect. All these pages point directly to ftp.m.o,
 http://www.mozilla.org/en-US/firefox/channel/#aurora
 http://www.mozilla.org/en-US/firefox/aurora/
 http://www.mozilla.org/en-US/firefox/all-aurora.html

They use firefox/nightly/latest-mozilla-aurora/ or latest-mozilla-aurora-l10n/, which have caching disabled or set to 0 seconds.

> We're talking about people that previously downloaded a stub installer, and
> then waited a day or more to run it, right?

Right, it's an edge case.

Comment 28

[Robert Strong (they/them - no direct email)]

Comment on attachment 807835 [details] [diff] [review]
update stub installer issuer

Looks fine.

Comment 29

[Robert Strong (they/them - no direct email)]

If we'd like to lessen the problems with people using an old stub the bouncer url could be changed though I don't think (fingers crossed) that will be too bad.

Comment 30

[[e-mail] (:bhearsum)]

OK, I've got a test updated path set-up as described by Brian's comment #24. I did a test from old cert -> new cert, then new cert -> new cert and everything succeeded without issue. The only thing of note is that both new cert MARs are exactly the same (to avoid me needing to do two builds) so the second download appears to be cached. I don't think this invalidates the test.

I'd really like a second set of eyes on this though. Nick or Brian, can one of you have a look as well? You'll need a mozilla-central nightly from the 23rd or earlier (I used https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013-09-23-03-02-04-mozilla-central/firefox-27.0a1.en-US.win32.installer.exe) and to point it at a different channel and update server:
* Set channel to nightly-cert-test (edit defaults/pref/channel-prefs.js in the appdir)
* Set app.update.url.override to https://aus4-dev.allizom.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

I've checked the update.xml served looks fine, and that the mar used (http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/new-cert-test/firefox-27.0a1.en-US.win32.complete.mar) contains files signed with the new cert. For the second update, if you're seeing the usual console messages from the update service about applying an update then you should be OK. I'll defer to Brian for verification of the other details.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 807835 [details] [diff] [review]
update stub installer issuer

Release managers, this patch needs to be landed on all branches at the same time as it needs to be synced up with me deploying changes to our signing servers.

Comment 33

[Brian R. Bondy [:bbondy]]

(In reply to Ben Hearsum [:bhearsum] from comment #30)
> OK, I've got a test updated path set-up as described by Brian's comment #24.
> I did a test from old cert -> new cert, then new cert -> new cert and
> everything succeeded without issue. The only thing of note is that both new
> cert MARs are exactly the same (to avoid me needing to do two builds) so the
> second download appears to be cached. I don't think this invalidates the
> test.

I think that's OK too.

> I'd really like a second set of eyes on this though. Nick or Brian, can one
> of you have a look as well? 

I ran through the test and it worked correctly. 

I also tested putting invalid registry info in case the registry becomes out of sync because of some bug somewhere, and confirmed the fallback is working correctly without errors visible to the user.

I also tested turning off the maintenance service and then verifying that the registry info is updated correctly.

---

By the way there is the ability to accept multiple different name/issuers (you can see how with the fallback key in the maintenance service with 2 subkeys, one with 0 and one with 1).  I don't think you need this, but thought I'd mentioned it.

Comment 34

[Lukas Blakk [:lsblakk] use ?needinfo]

From conversation in IRC it looks like we need to check with IT about ftp/CDN caching and then also with web production team about how the stub installer urls and bouncer link up via downloads.m.o -- bhearsum will be following up on those questions with those teams.

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 807835 [details] [diff] [review]
update stub installer issuer

Pushed to everywhere that does nightlies:
https://hg.mozilla.org/releases/mozilla-esr17/rev/2fb2d10898ef
https://hg.mozilla.org/releases/mozilla-esr24/rev/b079b31d322a
https://hg.mozilla.org/releases/mozilla-release/rev/7567981f3f0a
https://hg.mozilla.org/releases/mozilla-beta/rev/637bba8205e6
https://hg.mozilla.org/releases/mozilla-aurora/rev/06f04002d3fd
https://hg.mozilla.org/mozilla-central/rev/153aebb30387
https://hg.mozilla.org/releases/mozilla-b2g18/rev/9e45dd3f5428
https://hg.mozilla.org/projects/oak/rev/977573a12198
https://hg.mozilla.org/projects/ux/rev/710783606a1b
https://hg.mozilla.org/projects/elm/rev/153aebb30387

I've also restarted all of the nightly and release signing servers that use authenticode keys with the new certificates active.

I kicked a round of nightlies on mozilla-central, whose stub and full installers will both be signed with the new certificates.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/9e45dd3f5428

Comment 37

[[e-mail] (:bhearsum)]

I did an install from the Nightly stub installer this morning which worked fine. AFAIK this is validation of both the new certificates themselves and the cert pinning.

Comment 38

[Robert Strong (they/them - no direct email)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #29)
> If we'd like to lessen the problems with people using an old stub the
> bouncer url could be changed though I don't think (fingers crossed) that
> will be too bad.
After this landed the number of certificate attribute check failures went from an average of 5 per day across all channels with an average of 566,275 attempted installations per day to an average of 492 per day with an average of 700,679 attempted installations per day across all channels for the period from 9/27 through 10/28 though the certificate change only affected the number of certificate attribute failures on nightly, aurora, and beta.

On 10/29 the number of certificate attribute check failures increased again. For the period from 10/29 through 11/10 the number of certificate attribute check failures averaged 43,000 per day out of an average of 726,455 attempted installations per day and the certificate change affected the number of certificate attribute failures across all channels.

Date   # of failures
10/29	    63567
10/30	    71345
10/31	    57686
11/01	    51540
11/02	    42275
11/03	    39963
11/04	    40797
11/05	    37893
11/06	    37334
11/07	    34135
11/08	    31376
11/09	    27167
11/10	    23927

Comment 39

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #38)
> (In reply to Robert Strong [:rstrong] (do not email) from comment #29)
> > If we'd like to lessen the problems with people using an old stub the
> > bouncer url could be changed though I don't think (fingers crossed) that
> > will be too bad.
> After this landed the number of certificate attribute check failures went
> from an average of 5 per day across all channels with an average of 566,275
> attempted installations per day to an average of 492 per day with an average
> of 700,679 attempted installations per day across all channels for the
> period from 9/27 through 10/28 though the certificate change only affected
> the number of certificate attribute failures on nightly, aurora, and beta.
> 
> On 10/29 the number of certificate attribute check failures increased again.
> For the period from 10/29 through 11/10 the number of certificate attribute
> check failures averaged 43,000 per day out of an average of 726,455
> attempted installations per day and the certificate change affected the
> number of certificate attribute failures across all channels.
> 
> Date   # of failures
> 10/29	    63567
> 10/30	    71345
> 10/31	    57686
> 11/01	    51540
> 11/02	    42275
> 11/03	    39963
> 11/04	    40797
> 11/05	    37893
> 11/06	    37334
> 11/07	    34135
> 11/08	    31376
> 11/09	    27167
> 11/10	    23927

So, uh, this seems really really bad. I filed bug 938117 to investigate this.

Comment 40

[:Gijs (out for now; he/him)]

https://hg.mozilla.org/mozilla-central/rev/710783606a1b