Automate GPG signature verification

RESOLVED FIXED

Status

defect
P3
normal
RESOLVED FIXED
12 years ago
6 years ago

People

(Reporter: joduinn, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [simple])

Attachments

(2 attachments, 6 obsolete attachments)

The signing step currently does:
- GPG signing for linux
- GPG signing for mac (bug#400296 is about adding signing to Mac builds)
- Authenticode and GPG signing for win32

For verification, we currently do some simple manual signature verification for Authenticode only. Is there a way to automate this verification, and not do it by hand anymore?

Also, GPG verification needs to be established. Doing this in an automated way would be best.

Originally, was thinking this signature verification would happen on the signing machine. However, as signature verification only needs public keys, I'm now thinking it could be run on regular build machines with access to public keys. This could even help detect if the signing machine had been compromised.

In automation sequence, I guess signature verification has to happen sometime after signing and before pushing to mirrors.
Also see bug 372765
Assignee: nobody → rhelmer
Priority: -- → P2
Status: NEW → ASSIGNED
This verifies that:

1) all (bz2, dmg, exe) files in releases area match the files in candidates area (uses sha1 sums to compare), and have valid GPG detached signatures 
2) all win32 installer and internal files have signatures

Using chktrust.exe from Mono on Linux as suggested in 372765#c3 but getting:
"couldn't find the certificate that signed the file"
I have imported all the necessary CA certs, but might need our public cert too, I'll investigate this more later. 

For now, this script just verifies that the win32 installer and every EXE/DLL that is not one of the NSS DLLs has been signed with authenticode, and that the contents of the signed win32 installer match the contents of the equivalent file in the candidates area. 

There doesn't seem to be an easy way to say "file a matches file b without authenticode" apart from a full unpack/compare the contents, so that's what this does. 

This script is intended to run in the private staging area before release (so it can find the gpg public key file "KEY"), and currently needs absolute paths for it's two required arguments: release_dir and candidates_dir.

Still need to support xpi and mar files, but since these are not signed it is just a matter of getting the candidates -> release name mapping done correctly. I have only tested this on a small subset of files, I intend to try with the full 3.0b2 release this week.
(In reply to comment #0)
> In automation sequence, I guess signature verification has to happen sometime
> after signing and before pushing to mirrors.

I think this should happen last, literally the last thing before the final rsync to the releases area; it's be post-"stage-merged".

This script was written to discover files and then verify them; however I think it'd be easy and also very useful to have it make sure that everything that *should* be present is (all locales present and accounted for, all expected filetypes, expected filenames (3.0b2 verus 3.0 Beta 2), valid ownership and permissions, etc.). The current script would not alert us if there was a locale missing.

There is a script, verify-locales, which runs on the stage-signed area before signing, but a couple of manual steps happen in between verify-locales running and us being ready to ship:

1) stage-signed area is actually signed 
2) everything is copied into stage-merged area

Maybe this makes sense as a separate script, but there's certainly a lot of overlap. Any thoughts? 
Whiteboard: proof-of-concept up, trolling for feedback
Priority: P2 → P3
Whiteboard: proof-of-concept up, trolling for feedback → testing
Whiteboard: testing
I don't have time to finish up and integrate this right now, returning to the pool for the time being.
Assignee: rhelmer → nobody
Status: ASSIGNED → NEW
Triaged to Future.
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
We do the authenticode part of this now, since bug 470146 landed.
Summary: Automate signature verification → Automate GPG signature verification
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Whiteboard: [simple]
Assignee: nobody → raliiev
Status: NEW → ASSIGNED
Attached is a simplified (gpg only) script.

It can be used as:

python tools/release/signing/verify-gpg.py -k KEY -x /contrib/ -x /contrib-localized/ /path/to/gpg-signed-dir

On every run it creates a fresh homedir for gpg and imports the key.

Simple output is:

tools/release/signing/verify-gpg.py -k test1/KEY  -x /contrib-localized/ test1 
------
Verifying test1/1.bz2... OK
Verifying test1/1.dmg... OK
Verifying test1/1.exe... OK
Verifying test1/contrib/1.bz2... Signature doesn't exist
Verifying test1/contrib/1.dmg... Signature doesn't exist
Verifying test1/contrib/1.exe... Signature doesn't exist
FAILED
------
exit 1

Or:

tools/release/signing/verify-gpg.py -k test1/KEY \
  -x /contrib/ -x /contrib-localized/ test1 
--------
Verifying test1/1.bz2... OK
Verifying test1/1.dmg... OK
Verifying test1/1.exe... OK
--------
exit 0

The only thing I still not sure is how to run this script.

1. Manually by adding a makefile (or a target for tools/release/signing/Makefile) which downloads the files, and then runs the script

2. As a buildbot BuildStep, part of ReleaseFactory.

Disk space requirement in both cases is the same.
Attachment #294638 - Attachment is obsolete: true
Attachment #432748 - Flags: feedback?(bhearsum)
ReleaseGPGVerification factory
Posted patch buildbot-configs changes (obsolete) — Splinter Review
I've managed to test this bug today. The main problem for staging is absence of the signing step. Another problem is limited access to staging-stage and stage.

I used the following work around:

1. Reconfig the master.
2. Login to one  of the attached linux slaves
3. rsync 3.6.2 release to a temporary directory (from public rsync servers)
4. Push files to staging-stage (/home/ftp/pub...)
5. Force gpg_verification step (release_config was adjusted for 3.6.2 build3)

Everything went smooth.
Refreshed context
Attachment #433304 - Attachment is obsolete: true
Attachment #434855 - Flags: review?(catlee)
sort files
Attachment #432748 - Attachment is obsolete: true
Attachment #434857 - Flags: review?(catlee)
Attachment #432748 - Flags: feedback?(bhearsum)
Attachment #433305 - Flags: review?(catlee)
I realize the reviews were just moved away from me, but I still want to say that I don't think this should be run in a Buildbot factory. I think it makes much more sense to run it with the other signing verifications, on the signing machine. Adding another target to the signing Makefile, and having it run as part of the 'postsign' metatarget would make sense.

What do you guys think?
Yeah, I'd agree it belongs as part of the signing process.  We should check that the GPG signatures are valid for the public key on our FTP site and not rely on public/private keys on the machine though.
(In reply to comment #15)
> Yeah, I'd agree it belongs as part of the signing process.  We should check
> that the GPG signatures are valid for the public key on our FTP site and not
> rely on public/private keys on the machine though.

Note that the key gets put in $(WORKDIR)/signed-build$(BUILD)/KEY as part of the signing process, so there's not even a need to pull it from FTP.
(In reply to comment #16)
> Note that the key gets put in $(WORKDIR)/signed-build$(BUILD)/KEY as part of
> the signing process, so there's not even a need to pull it from FTP.

I intentionally used a public source for downloading published GPG key. Some kind of paranoia. :)

The main idea of this bug is GPG checking, but the secondary one was detecting intrusions to the keymaster machine. To implement second one, it will be better to use "clear room" environment: files and signatures from FTP master, public key from another source. Using slaves for such task is almost ideal.

If we decide to not use slaves (and buildbot) for this, it won't be hard to add a target to the Makefile.
Comment on attachment 434855 [details] [diff] [review]
ReleaseGPGVerification factory

rather have this in the signing Makefile for now, as discussed on irc.
Attachment #434855 - Flags: review?(catlee) → review-
Attachment #433305 - Flags: review?(catlee) → review-
Attachment #434857 - Flags: review?(catlee) → review+
Posted patch Makefile changes (obsolete) — Splinter Review
(In reply to comment #18)
> (From update of attachment 434855 [details] [diff] [review])
> rather have this in the signing Makefile for now, as discussed on irc.

Please review the attached Makefile changes
Attachment #433305 - Attachment is obsolete: true
Attachment #434855 - Attachment is obsolete: true
Attachment #435902 - Flags: review?(catlee)
I just tested GPG verification for 3.6.2 and it took less than 2 minutes on Linux VM (512M RAM, 20% of 3GHz CPU). It would be great if somebody could test this script on keymaster machine.
Attachment #435902 - Attachment is obsolete: true
Attachment #436189 - Flags: review?(catlee)
Attachment #435902 - Flags: review?(catlee)
Attachment #436189 - Flags: review?(catlee) → review+
Attachment #434857 - Flags: checked-in?
Attachment #436189 - Flags: checked-in?
Comment on attachment 434857 [details]
tools/release/signing/verify-gpg.py

changeset:   566:f5429d8f3021
Attachment #434857 - Flags: checked-in? → checked-in+
Comment on attachment 436189 [details] [diff] [review]
Makefile changes

changeset:   566:f5429d8f3021
Attachment #436189 - Flags: checked-in? → checked-in+
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.