409477 - Automate GPG signature verification

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

The signing step currently does:
- GPG signing for linux
- GPG signing for mac (bug#400296 is about adding signing to Mac builds)
- Authenticode and GPG signing for win32

For verification, we currently do some simple manual signature verification for Authenticode only. Is there a way to automate this verification, and not do it by hand anymore?

Also, GPG verification needs to be established. Doing this in an automated way would be best.

Originally, was thinking this signature verification would happen on the signing machine. However, as signature verification only needs public keys, I'm now thinking it could be run on regular build machines with access to public keys. This could even help detect if the signing machine had been compromised.

In automation sequence, I guess signature verification has to happen sometime after signing and before pushing to mirrors.

Comment 1

[Robert Helmer [:rhelmer]]

Also see bug 372765

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: WIP, quick first stab at signature verification

This verifies that:

1) all (bz2, dmg, exe) files in releases area match the files in candidates area (uses sha1 sums to compare), and have valid GPG detached signatures 
2) all win32 installer and internal files have signatures

Using chktrust.exe from Mono on Linux as suggested in 372765#c3 but getting:
"couldn't find the certificate that signed the file"
I have imported all the necessary CA certs, but might need our public cert too, I'll investigate this more later. 

For now, this script just verifies that the win32 installer and every EXE/DLL that is not one of the NSS DLLs has been signed with authenticode, and that the contents of the signed win32 installer match the contents of the equivalent file in the candidates area. 

There doesn't seem to be an easy way to say "file a matches file b without authenticode" apart from a full unpack/compare the contents, so that's what this does. 

This script is intended to run in the private staging area before release (so it can find the gpg public key file "KEY"), and currently needs absolute paths for it's two required arguments: release_dir and candidates_dir.

Still need to support xpi and mar files, but since these are not signed it is just a matter of getting the candidates -> release name mapping done correctly. I have only tested this on a small subset of files, I intend to try with the full 3.0b2 release this week.

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> In automation sequence, I guess signature verification has to happen sometime
> after signing and before pushing to mirrors.

I think this should happen last, literally the last thing before the final rsync to the releases area; it's be post-"stage-merged".

This script was written to discover files and then verify them; however I think it'd be easy and also very useful to have it make sure that everything that *should* be present is (all locales present and accounted for, all expected filetypes, expected filenames (3.0b2 verus 3.0 Beta 2), valid ownership and permissions, etc.). The current script would not alert us if there was a locale missing.

There is a script, verify-locales, which runs on the stage-signed area before signing, but a couple of manual steps happen in between verify-locales running and us being ready to ship:

1) stage-signed area is actually signed 
2) everything is copied into stage-merged area

Maybe this makes sense as a separate script, but there's certainly a lot of overlap. Any thoughts? 

Comment 4

[Robert Helmer [:rhelmer]]

I don't have time to finish up and integrate this right now, returning to the pool for the time being.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Triaged to Future.

Comment 6

[[e-mail] (:bhearsum)]

We do the authenticode part of this now, since bug 470146 landed.

Comment 7

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 8

[Rail Aliiev [:rail]]

Attachment: tools/release/signing/verify-gpg.py

Attached is a simplified (gpg only) script.

It can be used as:

python tools/release/signing/verify-gpg.py -k KEY -x /contrib/ -x /contrib-localized/ /path/to/gpg-signed-dir

On every run it creates a fresh homedir for gpg and imports the key.

Simple output is:

tools/release/signing/verify-gpg.py -k test1/KEY  -x /contrib-localized/ test1 
------
Verifying test1/1.bz2... OK
Verifying test1/1.dmg... OK
Verifying test1/1.exe... OK
Verifying test1/contrib/1.bz2... Signature doesn't exist
Verifying test1/contrib/1.dmg... Signature doesn't exist
Verifying test1/contrib/1.exe... Signature doesn't exist
FAILED
------
exit 1

Or:

tools/release/signing/verify-gpg.py -k test1/KEY \
  -x /contrib/ -x /contrib-localized/ test1 
--------
Verifying test1/1.bz2... OK
Verifying test1/1.dmg... OK
Verifying test1/1.exe... OK
--------
exit 0

The only thing I still not sure is how to run this script.

1. Manually by adding a makefile (or a target for tools/release/signing/Makefile) which downloads the files, and then runs the script

2. As a buildbot BuildStep, part of ReleaseFactory.

Disk space requirement in both cases is the same.

Comment 9

[Rail Aliiev [:rail]]

Attachment: ReleaseGPGVerification factory

ReleaseGPGVerification factory

Comment 10

[Rail Aliiev [:rail]]

Attachment: buildbot-configs changes

Comment 11

[Rail Aliiev [:rail]]

I've managed to test this bug today. The main problem for staging is absence of the signing step. Another problem is limited access to staging-stage and stage.

I used the following work around:

1. Reconfig the master.
2. Login to one  of the attached linux slaves
3. rsync 3.6.2 release to a temporary directory (from public rsync servers)
4. Push files to staging-stage (/home/ftp/pub...)
5. Force gpg_verification step (release_config was adjusted for 3.6.2 build3)

Everything went smooth.

Comment 12

[Rail Aliiev [:rail]]

Attachment: ReleaseGPGVerification factory

Refreshed context

Comment 13

[Rail Aliiev [:rail]]

Attachment: tools/release/signing/verify-gpg.py

sort files

Comment 14

[[e-mail] (:bhearsum)]

I realize the reviews were just moved away from me, but I still want to say that I don't think this should be run in a Buildbot factory. I think it makes much more sense to run it with the other signing verifications, on the signing machine. Adding another target to the signing Makefile, and having it run as part of the 'postsign' metatarget would make sense.

What do you guys think?

Comment 15

[Chris AtLee [:catlee]]

Yeah, I'd agree it belongs as part of the signing process.  We should check that the GPG signatures are valid for the public key on our FTP site and not rely on public/private keys on the machine though.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to comment #15)
> Yeah, I'd agree it belongs as part of the signing process.  We should check
> that the GPG signatures are valid for the public key on our FTP site and not
> rely on public/private keys on the machine though.

Note that the key gets put in $(WORKDIR)/signed-build$(BUILD)/KEY as part of the signing process, so there's not even a need to pull it from FTP.

Comment 17

[Rail Aliiev [:rail]]

(In reply to comment #16)
> Note that the key gets put in $(WORKDIR)/signed-build$(BUILD)/KEY as part of
> the signing process, so there's not even a need to pull it from FTP.

I intentionally used a public source for downloading published GPG key. Some kind of paranoia. :)

The main idea of this bug is GPG checking, but the secondary one was detecting intrusions to the keymaster machine. To implement second one, it will be better to use "clear room" environment: files and signatures from FTP master, public key from another source. Using slaves for such task is almost ideal.

If we decide to not use slaves (and buildbot) for this, it won't be hard to add a target to the Makefile.

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 434855 [details] [diff] [review]
ReleaseGPGVerification factory

rather have this in the signing Makefile for now, as discussed on irc.

Comment 19

[Rail Aliiev [:rail]]

Attachment: Makefile changes

(In reply to comment #18)
> (From update of attachment 434855 [details] [diff] [review])
> rather have this in the signing Makefile for now, as discussed on irc.

Please review the attached Makefile changes

Comment 20

[Rail Aliiev [:rail]]

I just tested GPG verification for 3.6.2 and it took less than 2 minutes on Linux VM (512M RAM, 20% of 3GHz CPU). It would be great if somebody could test this script on keymaster machine.

Comment 21

[Rail Aliiev [:rail]]

Attachment: Makefile changes

Comment 22

[Chris AtLee [:catlee]]

Comment on attachment 434857 [details]
tools/release/signing/verify-gpg.py

changeset:   566:f5429d8f3021

Comment 23

[Chris AtLee [:catlee]]

Comment on attachment 436189 [details] [diff] [review]
Makefile changes

changeset:   566:f5429d8f3021