1284484 - refresh Windows XP installer before SHA-1 certificate expires on September 21, 2016

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[[e-mail] (:bhearsum)]

Right now we've got new installs for XP users going through an old 43.0 installer because that was the last release we signed with SHA-1 certificates. Our SHA-1 cert expires in September, and we can improve the experience for these users if we do a one-off manual resign of the latest installers before we lose the ability to. IIRC, we need to resign the stub and full installers, but I could be wrong about that.

The root bug where we dealt with this last time is https://bugzilla.mozilla.org/show_bug.cgi?id=1079858.

Comment 1

[Rail Aliiev [:rail]]

Just to make sure. We need to resign only the installer, not the files inside it. Correct?

The workflow would be something like:

1) resign the installers (maybe using the partner repack script?) for release, beta and esr45
2) publish them to a separate directory under /releases
3) create bouncer entries for them
4) update and deploy https://github.com/mozilla-services/go-bouncer/blob/master/handlers.go#L29-L31

Does it sound correct?

Also, would be great to try this method ASAP to make sure it works, so we are not in rush by the end of the term.

Comment 2

[Rail Aliiev [:rail]]

Doing the same for TB would be karma++ :)

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #1)
> Just to make sure. We need to resign only the installer, not the files
> inside it. Correct?

That's my recollection. The easiest way to be sure is probably to inspect the current installer we give to XP users.

Comment 4

[[e-mail] (:bhearsum)]

We should prioritize this soon. Turns out the current SHA-1 watershed on Nightly and Aurora is busted because the MARs got deleted, eg: https://mozilla-nightly-updates.s3.amazonaws.com/mozilla-central/20151209095500/Firefox-mozilla-central-45.0a1-win32-ja-JP-mac-20151208030212-20151209095500.partial.mar?versionId=rnBBKoKAnOsWAibq1wd2GwCT0sNExZKC

I don't think that this doesn't affect new installs (they get the 43.0 installer and then update to latest), just people updating from older versions.

Comment 5

[Rail Aliiev [:rail]]

I hacked the partner repack script so it only removes the signature and resigns the installers:

--- a/partner-repacks.py
+++ b/partner-repacks.py
@@ -522,11 +522,11 @@ class RepackBase(object):
         self.announceStart()
         os.chdir(self.working_dir)
         self.unpackBuild()
-        self.copyFiles()
-        self.addPadding()
-        if self.signing_command and self.internal_signing_formats:
-            self.internallySignBuild()
-        self.repackBuild()
+        # self.copyFiles()
+        # self.addPadding()
+        # if self.signing_command and self.internal_signing_formats:
+        #     self.internallySignBuild()
+        # self.repackBuild()
         if self.signing_command and self.external_signing_formats:
             self.externallySignBuild()
         self.stage()
@@ -667,7 +667,7 @@ class RepackWinBase(RepackBase):
 class RepackWin(RepackWinBase):
     def __init__(self, build, partner_dir, build_dir, final_dir,
                  ftp_platform, repack_info, signing_command,
-                 external_signing_formats=['gpg', 'sha2signcode'], **kwargs):
+                 external_signing_formats=['gpg', 'osslsigncode'], **kwargs):
         super(RepackWin, self).__init__(build, partner_dir, build_dir,
                                         final_dir,
                                         ftp_platform, repack_info,
@@ -679,7 +679,7 @@ class RepackWin(RepackWinBase):
 class RepackWin64(RepackWinBase):
     def __init__(self, build, partner_dir, build_dir, final_dir,
                  ftp_platform, repack_info, signing_command,
-                 external_signing_formats=['gpg', 'sha2signcode'], **kwargs):
+                 external_signing_formats=['gpg', 'osslsigncode'], **kwargs):
         super(RepackWin64, self).__init__(build, partner_dir, build_dir,
                                         final_dir,
                                         ftp_platform, repack_info,


My current repacks.cfg is:

$ cat ../mozilla-sha1/desktop/mozilla-sha1/repack.cfg 
linux-i686=false
linux-x86_64=false
mac=false
win32=true
win64=true
locales="ach af an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ja-JP-mac kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW"
output_dir="%(platform)s-sha1/%(locale)s"

## Upload params
s3cfg=/builds/release-s3cfg
bucket="net-mozaws-prod-delivery-firefox"
upload_to_candidates=true

I also added Firefox-beta-sha1 product to bouncer and set the location pointing to repacks in the 49.0b9 candidates directory. This way we patch bouncer only once and use this product when we refresh the binaries again. The same will be created for release and esr.

After I finished running repacks we need to verify:

1) bouncer works as expected
2) verify only 1 signature is used and the cert is not the sha2 cert
3) it launches on winxp (will need QA help here)

Comment 6

[Rail Aliiev [:rail]]

The list of locales was generated by Quick and Powerful Shell! :)

curl http://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/l10n_changesets.txt | awk '{print $1} END {print "en-US"}' | grep -v ^$ | sort | xargs

Comment 7

[Rail Aliiev [:rail]]

Bogdan,

May I ask you to spot check if the installers from the following locations can be launched on Windows XP SP2 (it is important to not have SP3 installed):


https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/
https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/

2-3 locales per platform would be enough to verify I signed them properly.

If you can check these some time next week, I'd appreciate it!

Thank you in advance.

Comment 8

[Rail Aliiev [:rail]]

I just checked, http://download.mozilla.org/?product=Firefox-beta-sha1&os=win&lang=en-US and friends WFM.

Comment 9

[Lawrence Mandel [:lmandel] (use needinfo)]

ni Andrei as well in case someone else can help with verification

Comment 10

[Bogdan Maris, Desktop Test Engineering]

(In reply to Rail Aliiev [:rail] from comment #7)
> Bogdan,
> 
> May I ask you to spot check if the installers from the following locations
> can be launched on Windows XP SP2 (it is important to not have SP3
> installed):
> 
> 
> https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/
> win32-sha1/
> https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/
> win64-sha1/
> 
> 2-3 locales per platform would be enough to verify I signed them properly.
> 
> If you can check these some time next week, I'd appreciate it!
> 
> Thank you in advance.

Sure thing, sorry for the delay, I was a bit busy.

Testing was done on two different machines with Windows XP 32bit SP2 and Windows XP 64bit SP2.
I grabbed 4 locales (ar, de, fr and ja) and I confirm that the builds start successfully.

- Windows XP Professional (5.1, Build 2600):
Digital signature:
> Issued to: Mozilla Corporation
> Issued by: DigiCert Sha2 Assured ID Code Signing CA
> Valid from 7/9/2015 to 7/13/2018

With XP 64-bit version though, the 64-bit builds can't be installed: "Sorry, Firefox can't be installed. This version of Firefox requires Microsoft Windows 7 x64 or newer." I am able though to install 32-bit builds and they work just fine.

- Windows XP Professional x64 Edition (5.2, Build 3790):
Digital signature:
> Issued to: Mozilla Corporation
> Issued by: DigiCert Sha2 Assured ID Code Signing CA
> Valid from 7/9/2015 to 7/13/2018

Comment 11

[Rail Aliiev [:rail]]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #10)
> Sure thing, sorry for the delay, I was a bit busy.

No worries, at all. I know that releases are priority #1 now. :)
 
> Digital signature:
> > Issued to: Mozilla Corporation
> > Issued by: DigiCert Sha2 Assured ID Code Signing CA
> > Valid from 7/9/2015 to 7/13/2018

Is this the signature of firefox.exe or the installer (Firefox Setup 48.0b9.exe)?

firefox.exe should have the signature you mentioned, but the installer should be signed using by a cert issued by "DigiCert Assured ID Code Signing CA-1". Can you verify this?

Can you also paste the exact URL of the file you tested? I just checked https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/ja/Firefox%20Setup%2048.0b9.exe and see the following signature (no "Sha2" in the name):

http://people.mozilla.org/~raliiev/sattap/2fcea7f9.png


> With XP 64-bit version though, the 64-bit builds can't be installed: "Sorry,
> Firefox can't be installed. This version of Firefox requires Microsoft
> Windows 7 x64 or newer." I am able though to install 32-bit builds and they
> work just fine.
> - Windows XP Professional x64 Edition (5.2, Build 3790):
> Digital signature:
> > Issued to: Mozilla Corporation
> > Issued by: DigiCert Sha2 Assured ID Code Signing CA
> > Valid from 7/9/2015 to 7/13/2018

Hmm. The same here, can you send the URLs of the files you tested?

Comment 12

[Rail Aliiev [:rail]]

Note for myself. We probably want to stop serving stub installers for these users, because the stub installer checks the signature of the full installer against a pinned value [1][2]  and it won't be able to verify the SHA1 cert.

I'll file a bug to change bouncer [3] code serving stub for XP and Co.

1. https://dxr.mozilla.org/mozilla-central/source/browser/branding/official/branding.nsi#28
2. https://dxr.mozilla.org/mozilla-central/source/browser/installer/windows/nsis/stub.nsi#1566
3. https://github.com/mozilla-services/go-bouncer/blob/master/handlers.go#L118-L121

Comment 13

[Bogdan Maris, Desktop Test Engineering]

(In reply to Rail Aliiev [:rail] from comment #11)
> (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #10)
> > Sure thing, sorry for the delay, I was a bit busy.
> 
> No worries, at all. I know that releases are priority #1 now. :)
>  
> > Digital signature:
> > > Issued to: Mozilla Corporation
> > > Issued by: DigiCert Sha2 Assured ID Code Signing CA
> > > Valid from 7/9/2015 to 7/13/2018
> 
> Is this the signature of firefox.exe or the installer (Firefox Setup
> 48.0b9.exe)?
> 
> firefox.exe should have the signature you mentioned, but the installer
> should be signed using by a cert issued by "DigiCert Assured ID Code Signing
> CA-1". Can you verify this?
> 
> Can you also paste the exact URL of the file you tested? I just checked
> https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/
> win32-sha1/ja/Firefox%20Setup%2048.0b9.exe and see the following signature
> (no "Sha2" in the name):
> 
> http://people.mozilla.org/~raliiev/sattap/2fcea7f9.png
 
Yep, I checked the signature firefox.exe, my bad.
 
I confirm that the certificate is "DigiCert Assured ID Code Signing CA-1" for the following installers:
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/ar/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/de/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/fr/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win32-sha1/ja/Firefox%20Setup%2048.0b9.exe


> > With XP 64-bit version though, the 64-bit builds can't be installed: "Sorry,
> > Firefox can't be installed. This version of Firefox requires Microsoft
> > Windows 7 x64 or newer." I am able though to install 32-bit builds and they
> > work just fine.
> > - Windows XP Professional x64 Edition (5.2, Build 3790):
> > Digital signature:
> > > Issued to: Mozilla Corporation
> > > Issued by: DigiCert Sha2 Assured ID Code Signing CA
> > > Valid from 7/9/2015 to 7/13/2018
> 
> Hmm. The same here, can you send the URLs of the files you tested?

Sure thing, I tested with the same as above but vrom win64-sha1 plus ro and en-US:
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/ar/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/de/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/fr/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/ja/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/ro/Firefox%20Setup%2048.0b9.exe
 - https://ftp.mozilla.org/pub/firefox/candidates/48.0b9-candidates/build1/win64-sha1/en-US/Firefox%20Setup%2048.0b9.exe

A thing to note here is the cert is the correct one for these installers as well "DigiCert Assured ID Code Signing CA-1"
Image: http://imgur.com/a/kxDA1

Comment 14

[Rail Aliiev [:rail]]

Oh, sweet! This is what I wanted to see! :)

I'm not quite sure what to do with win64. Probably nothing to do, poor XP users. :/

Comment 15

[Bogdan Maris, Desktop Test Engineering]

Can we just serve users that use XP 64bit with 32bit builds for now? Also maybe an article on SUMO would help, in which we advise them to use 32bit installers instead of 64bit.

Comment 16

[Rail Aliiev [:rail]]

(In reply to Bogdan Maris, QA [:bogdan_maris][PTO 08-22 Aug] from comment #15)
> Can we just serve users that use XP 64bit with 32bit builds for now? 

Per https://www.mozilla.org/en-US/firefox/48.0/system-requirements/ Win7 is required for 64-bit.

Comment 17

[Rail Aliiev [:rail]]

Beta points to 49.0b8 now.

Comment 18

[Rail Aliiev [:rail]]

> refresh Windows XP installer before SHA-1 certificate expires on September 21, 2016

sure, today is Sep 20 :)