1335667 - Validate all downloaded sources when building GCC

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1335667 - Use `set -e` instead of manual `exit 1`.

Review commit: https://reviewboard.mozilla.org/r/108700/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/108700/

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1335667 - Validate all downloaded sources when building GCC.

The GCC and binutils tarballs are GPG signed upstream, so we can just
check the signature against the corresponding GPG keys.

I validated the GPG keys by:
- Creating a fresh keyring.
- Importing the keys with gpg --receive-key.
- Importing my own GPG public key in that keyring.
- Importing the gpg keys that the PGP pathfinder told me were on the path
  to those keys (which weren't directly in their keyring, so I had to
  manually find some steps first).
- Using `gpg --check-sigs` to validate that the all those keys I got are
  the right ones.
- Exporting the GCC and binutils upstream keys with `gpg --export
  --armor`.

Sadly, the GCC prerequisites are not signed, so I had to:
- Download the files from ftp.gnu.org.
- Download the corresponding files from snapshot.debian.org.
- Compare the raw files when possible, or the uncompressed (not extracted)
  files (when, thankfully, they matched).
- For gmp, compare the contents, which differed because some files were
  removed in the Debian source, and autoconf re-run.
- Validate those snapshot.debian.org files checksums against the
  checksums in the corresponding Sources.bz2/xz files.
- Validate the Sources.bz2/xz checksums against the corresponding InRelease
  files.
- Validate the InRelease files GPG signatures against the Debian
  archives keyring.

snapshot.debian.org didn't, however, have the source of version 0.8.1 of
MPC. And the first version ever GPG signed on
http://www.multiprecision.org was 0.8.2. So this one was not
double-checked.

Review commit: https://reviewboard.mozilla.org/r/108702/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/108702/

Comment 3

[Gregory Szorc [:gps]]

Thank you for doing this.

Comment 4

[Nathan Froyd [:froydnj]]

Comment on attachment 8832370 [details]
Bug 1335667 - Use `set -e` instead of manual `exit 1`.

https://reviewboard.mozilla.org/r/108700/#review109982

So much better.

Comment 5

[Nathan Froyd [:froydnj]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

https://reviewboard.mozilla.org/r/108702/#review109986

Thanks for the thorough description of the steps you went through.  I noted prerequisites that have signatures from their original download sources; can we validate those as well?

::: build/unix/build-gcc/build-gcc.sh:22
(Diff revision 1)
> +$GPG --import $this_path/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
> +$GPG --import $this_path/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key

Please add a comment here as to which key is which, so if we ever have to change them, it's easy to know which ones to remove.

::: build/unix/build-gcc/prerequisites.sums:2
(Diff revision 1)
> +02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196  cloog-0.18.1.tar.gz
> +936162c0312886c21581002b79932829aa048cfaf9937c6265aeaa14f1cd1775  gmp-4.3.2.tar.bz2

GMP 4.3.2 appears to be signed on https://gmplib.org/download/gmp/archive/

::: build/unix/build-gcc/prerequisites.sums:5
(Diff revision 1)
> +02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196  cloog-0.18.1.tar.gz
> +936162c0312886c21581002b79932829aa048cfaf9937c6265aeaa14f1cd1775  gmp-4.3.2.tar.bz2
> +f4b3dbee9712850006e44f0db2103441ab3d13b406f77996d1df19ee89d11fb4  isl-0.12.2.tar.bz2
> +e664603757251fd8a352848276497a4c79b7f8b21fd8aedd5cc0598a38fee3e4  mpc-0.8.1.tar.gz
> +c7e75a08a8d49d2082e4caee1591a05d11b9d5627514e678f02d66a124bcf2ba  mpfr-2.4.2.tar.bz2

MPFR 2.4.2 appears to be signed on http://www.mpfr.org/mpfr-2.4.2/

Comment 6

[Mike Hommey [:glandium]]

(In reply to Nathan Froyd [:froydnj] from comment #5)
> ::: build/unix/build-gcc/prerequisites.sums:2
> (Diff revision 1)
> > +02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196  cloog-0.18.1.tar.gz
> > +936162c0312886c21581002b79932829aa048cfaf9937c6265aeaa14f1cd1775  gmp-4.3.2.tar.bz2
> 
> GMP 4.3.2 appears to be signed on https://gmplib.org/download/gmp/archive/

... except that's signed with a GPG key that is only signed by a revoked key and another key that only has a self-signature. Versions >= 5.1.0 are signed with a new key that is more verifiable.

I went ahead and used MPC 0.8.2 instead of 0.8.1 to have a GPG-validated source, it would probably make sense to do the same with GMP. Does 5.1.3 seem reasonable to you? For one, linux distros are usually using even newer versions for their builds of GCC...

Comment 7

[Nathan Froyd [:froydnj]]

(In reply to Mike Hommey [:glandium] from comment #6)
> (In reply to Nathan Froyd [:froydnj] from comment #5)
> > ::: build/unix/build-gcc/prerequisites.sums:2
> > (Diff revision 1)
> > > +02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196  cloog-0.18.1.tar.gz
> > > +936162c0312886c21581002b79932829aa048cfaf9937c6265aeaa14f1cd1775  gmp-4.3.2.tar.bz2
> > 
> > GMP 4.3.2 appears to be signed on https://gmplib.org/download/gmp/archive/
> 
> ... except that's signed with a GPG key that is only signed by a revoked key
> and another key that only has a self-signature. Versions >= 5.1.0 are signed
> with a new key that is more verifiable.

Doh.  I was wondering about that.

> I went ahead and used MPC 0.8.2 instead of 0.8.1 to have a GPG-validated
> source, it would probably make sense to do the same with GMP. Does 5.1.3
> seem reasonable to you? For one, linux distros are usually using even newer
> versions for their builds of GCC...

I think that's fine if everything builds and works OK.

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/108702/diff/1-2/

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/108702/diff/2-3/

Comment 10

[Mike Hommey [:glandium]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

This changed enough that a re-review is in order.

Note that we /could/ strip the key files to only contain the keys, and not all the signature material which is not useful to verify the GPG signatures of the tarballs. That would make them much smaller. https://davesteele.github.io/gpg/2014/09/20/anatomy-of-a-gpg-key/ is enlightening. I didn't find a "just-do-it" program that would do that, though.

Comment 11

[Gregory Szorc [:gps]]

Keep in mind that GPG signatures only validate authenticity. Since a single GPG key could sign multiple archives, hashes are a stronger guarantee of content than verifying GPG signatures. IMO we should pin hashes everywhere and strive for GPG verification where possible.

Comment 12

[Nathan Froyd [:froydnj]]

(In reply to Gregory Szorc [:gps] from comment #11)
> Keep in mind that GPG signatures only validate authenticity. Since a single
> GPG key could sign multiple archives, hashes are a stronger guarantee of
> content than verifying GPG signatures.

I don't follow this statement.  Can you explain?

Comment 13

[Mike Hommey [:glandium]]

Are you worried about compromised GPG keys or rogue upstreams signing new tarballs for old releases?

Comment 14

[Gregory Szorc [:gps]]

(In reply to Mike Hommey [:glandium] from comment #13)
> Are you worried about compromised GPG keys or rogue upstreams signing new
> tarballs for old releases?

Both.

We care about determinism: that things don't change out from under us. Verifying GPG signatures doesn't verify determinism: only that X signed something [and that you should be able to trust what was signed].

Comment 15

[Mike Hommey [:glandium]]

I wanted to say we might as well put the signatures in-tree instead of downloading them, but looking at the GCC signatures, they actually are using SHA-1. Which is better than MD5, but not much.

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/108702/diff/3-4/

Comment 17

[Mike Hommey [:glandium]]

Nathan, I did some serious changes in there, please look at it again.

(Also, it turns out that the previous round didn't actually build the versions of gmp/mpc downloaded from upstream, and this time, I'm actually sure it happens because all downloads are done by the script)

Comment 18

[Mike Hommey [:glandium]]

Oh, btw, I was bored, so I created a script to strip the GPG keys.

Comment 19

[Nathan Froyd [:froydnj]]

Comment on attachment 8832371 [details]
Bug 1335667 - Validate all downloaded sources when building GCC.

https://reviewboard.mozilla.org/r/108702/#review112000

::: build/unix/build-gcc/build-gcc.sh:71
(Diff revision 4)
> +  # Divert commands that download_prerequisites use
> +  ln() { :; }
> +  tar() { :; }
> +  sed() { :; }
> +  wget() {
> +    echo $1
> +  }
> +
> +  . ./contrib/download_prerequisites
> +) | while read url; do

Bravo.  Bravo.

Comment 20

[Nathan Froyd [:froydnj]]

Review has been completed.  Sorry for the delay.

Comment 21

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #15)
> I wanted to say we might as well put the signatures in-tree instead of
> downloading them, but looking at the GCC signatures, they actually are using
> SHA-1. Which is better than MD5, but not much.

If we put hashes of the files we're using in-tree then there wouldn't be much value in checking signatures, would there? Presumably we'd want to check the signatures before committing the hashes.

Comment 22

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #21)
> (In reply to Mike Hommey [:glandium] from comment #15)
> > I wanted to say we might as well put the signatures in-tree instead of
> > downloading them, but looking at the GCC signatures, they actually are using
> > SHA-1. Which is better than MD5, but not much.
> 
> If we put hashes of the files we're using in-tree then there wouldn't be
> much value in checking signatures, would there? Presumably we'd want to
> check the signatures before committing the hashes.

It actually adds a safety net that $whoever updates the GCC version won't have forgotten to GPG-validate the source tarball.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5af868e8b595
Use `set -e` instead of manual `exit 1`. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/4854cd37594f
Validate all downloaded sources when building GCC. r=froydnj

Comment 24

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5af868e8b595
https://hg.mozilla.org/mozilla-central/rev/4854cd37594f