408958 - Consolidate windows signing into one step

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Mike Schroepfer]

As I understand our windows signing process now:

1) linux/mac builds publish to /nightly/2.0.0.n-candidates/ while win32 publishes to /nightly/2.0.0.n-candidates/unsigned/
2) win32 builds are copied to signing server, EXE/DLL signed (but not installers), results pushed to /nightly/2.0.0.n-candidates/
3) QA tests and signs off, beta period happens, etc. from /nightly/2.0.0.n-candidates/
4) once beta period is over, all builds are copied from /nightly/2.0.0.n-candidates/ and renamed, copied to signing server, win32 installer signing and detached GPG signatures are created, and builds are pushed back to private staging area on stage.mozilla.org
5) builds are copied from stage.mozilla.org's private area to the /releases/2.0.0.n/ area

We should really look to consolidate this into:

1) build machines publish to /nightly/2.0.0.n-candidates/unsigned/
2) rename, copy all to signing server, sign, push back to /nightly/2.0.0.n-candidates/
3) QA tests and signs off, beta period happens etc. from /nightly/2.0.0.n-candidates/
4) builds are copied from /nightly/2.0.0.n-candidates/ to /releases/2.0.0.n/

Which has a couple of advantages:

1) Do don't ever have builds in the world where the internal files are signed but the installer is not
2) We eliminate any changes to the bit from "QA signoff" to final release.  Only thing that happens is a copy of a directory.  This speeds up the last step and prevents even the possibility of error
3) Signing is the only thing right now that is difficult to full automate - this helps reduce the number of manual steps from 2 to 1 in the release process

Comment 1

[Robert Helmer [:rhelmer]]

I think the shortest path here would be to modify the current process from:

* Tag/Source
* Build/Repack
* Sign
* Updates/Verify
* Stage

To:

* Tag/Source
* Build/Repack
* Stage/Sign
* Updates/Verify

I think that the simplest way to go would be to do the Stage (aka "rename files") before any signing, and then teach the signing, verify, update scripts to all know about long filenames ("win32/en-US/Firefox 2.x Setup.exe" versus "firefox-2.x-win32.installer.exe").

We currently have a private area that gets built up as the release goes along, until the end when a human comes along, signs, and copies the results of everything into "stage-merged", which is what is directly copied to the release area.

What we could do instead is treat a public area as "stage-merged" and build this up as the "Stage/Sign" and then "Update/Verify" steps above complete. This is what QA would use to test, and what we would copy directly to the releases area.

Comment 2

[Robert Helmer [:rhelmer]]

Nick points out that we could do the "staging" step per-platform as part of the "push" for each Build/Repack step, which makes it more like:

* Tag/Source
* Build/Repack
* Sign/l10nVerify
* Updates/Verify

Build/Repack would publish builds in the "win32/en-US/Firefox 2.x Setup.exe" form for all OSes, which is what QA would test. Once QA signed off, we'd be able to copy this area directly to the releases area.

In addition to moving the staging code out of it's own step and into something that's called by Build/Repack, we'd need to change the following tools:

* signing (sign-release.pl needs to understand long filenames, if it doesn't already)
* update generation needs to be configured for long filenames (I think this is just a PatcherConfig change)
* l10n and update verify tools need to understand long filenames

None of this seems too difficult.. I am touching a bunch of the related code in bug 415970 so I'd like to get that landed first.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Blockers fixed.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Support release layout for first signing

Step 1, teach the script that signs the internals of windows installers/updates how to deal with a release structure.

The main changes here are to recursively look for files to sign (relative to the given input dir), and to handle an output dir that is file dependent (and won't exist). This should support both the old flat structure of 2.0.0.x & 3.0.x and the release one of 3.1 - looks OK in first round of tests but planning to do some more.

Have been thinking of this setup & call with the releases layout:
$ rsync -a .../3.1b1-candidates/build1/ unsigned-build1/
$ mkdir signed-build1/
$ perl signing/sign-release.pl -o signed-build1 -M dir -d unsigned-build1/unsigned -a firefox --release ${TAG} 2>&1 | tee win32_signing_build1.log
# fix up windows-xpi here, depending on what comes out of l10n
$ rsync -av --exclude '*win*'

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 341262 [details] [diff] [review]
[checked in] Support release layout for first signing

>Index: sign-release.pl
>===================================================================

>+# A recursive list of files in the given dir, done this way
>+# because File::Find's global vars and callbacks are icky (TM)
>+
>+sub GetDirListing {

I had to step myself through this with pen and paper, but looks good!


This all looks fine....hopefully I will get a chance to test it before the release!

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> Created an attachment (id=341262) [details]
> Support release layout for first signing
> 
> Step 1, teach the script that signs the internals of windows installers/updates
> how to deal with a release structure.
> 
> The main changes here are to recursively look for files to sign (relative to
> the given input dir), and to handle an output dir that is file dependent (and
> won't exist). This should support both the old flat structure of 2.0.0.x &
> 3.0.x and the release one of 3.1 - looks OK in first round of tests but
> planning to do some more.
> 
> Have been thinking of this setup & call with the releases layout:
> $ rsync -a .../3.1b1-candidates/build1/ unsigned-build1/
> $ mkdir signed-build1/
> $ perl signing/sign-release.pl -o signed-build1 -M dir -d
> unsigned-build1/unsigned -a firefox --release ${TAG} 2>&1 | tee
> win32_signing_build1.log
> # fix up windows-xpi here, depending on what comes out of l10n
> $ rsync -av --exclude '*win*'

This seems like a fine way to do things. Obviously, we need to call sign-release afterwards, though.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 341262 [details] [diff] [review]
[checked in] Support release layout for first signing

Checked in:
new revision: 1.24; previous revision: 1.23

(In reply to comment #5)
> >+sub GetDirListing {
> I had to step myself through this with pen and paper, but looks good!

We could revisit this and convert it to File::Find if you prefer, would probably improve readability.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Dump some variable values, suppress 7z/mar packing output

Putting this up so I don't lose it, it was handy for testing.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Simple script to download and rename builds

This creates a directory layout in the release style from a flat candidates dir. Afterwards we have

unsigned-build1/linux-i686/de/firefox-3.0.3.tar.bz2
unsigned-build1/linux-i686/en-US/firefox-3.0.3.tar.bz2
unsigned-build1/mac/de/Firefox 3.0.3.dmg
unsigned-build1/mac/en-US/Firefox 3.0.3.dmg
unsigned-build1/unsigned/update/win32/de/firefox-3.0.3.complete.mar
unsigned-build1/unsigned/update/win32/en-US/firefox-3.0.3.complete.mar
unsigned-build1/unsigned/win32/de/Firefox Setup 3.0.3.exe
unsigned-build1/unsigned/win32/en-US/Firefox Setup 3.0.3.exe
unsigned-build1/update/linux-i686/de/firefox-3.0.3.complete.mar
unsigned-build1/update/linux-i686/en-US/firefox-3.0.3.complete.mar
unsigned-build1/update/mac/de/firefox-3.0.3.complete.mar
unsigned-build1/update/mac/en-US/firefox-3.0.3.complete.mar

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #7)
> (From update of attachment 341262 [details] [diff] [review])
> Checked in:
> new revision: 1.24; previous revision: 1.23

Needed a slight fix to this, in GetDirListing, the arg checking needs to refer to $dir and not $args{'dir'}. That's rev 1.25.

Comment 11

[[e-mail] (:bhearsum)]

Hey Nick,

I worked through the new signing instructions..Everything looks good, the only tweak I made is to use 'tee -a' for internals signing logs so we don't wipe out the en-US stuff when we go back and sign locales.

Thanks so much!

Comment 12

[[e-mail] (:bhearsum)]

Turns out the $NEW_WIN32_EXE_REGEX doesn't support Beta/Alpha/RC filenames :(. I had to locally patch sign-release.pl like so to get things to work. I'll post it soon.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch to make alphas/betas/rcs work

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 342263 [details] [diff] [review]
[checked in] patch to make alphas/betas/rcs work

On second thought, I think this will be OK. If there's an obvious way to clean-up this up feel free to point them out though!

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 342263 [details] [diff] [review]
[checked in] patch to make alphas/betas/rcs work

Checking in sign-release.pl;
/mofo/release/signing/tools/sign-release.pl,v  <--  sign-release.pl
new revision: 1.26; previous revision: 1.25
done

Comment 16

[[e-mail] (:bhearsum)]

Hmmm, is there anything else to do here?

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Can't think of anything, and we can fix the other bugs with signing (like potential automation, supporting hg) elsewhere.