377781 - Generate new public gpg key, existing one has expired

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Both the KEY file and gpg say that the existing PGP sig expired 2006-09-29, so we should renew/regenerate as appropriate.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From offline discussion: As this is the day before the 2.0.0.4 release, we're not going to change this right now.

However, we do want to fix this for 2.0.0.5, hence filed tracking bug and assigned to me. Will start working on this after 2.0.0.4 ships.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

We have copies of the public key available at:
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/1.5/KEY
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0/KEY
...
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/granparadiso/alpha6/KEY

These copies of public key are put here as part of the release process, so a copy is placed with each new release. A quick spot check of a few showed them all to be identical.


Some quick websurfing also found copies at:
http://pgp.mit.edu:11371/pks/lookup?search=mozilla.org&op=index
http://keyserver.veridis.com:11371/search?q=mozilla.org&searchformsubmit=Search
(...keyserver.net redirects to keyserver.veridis.com...)

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Discovered that GPG allows you to extend/modify the expiry date of a key. On my machine I did the following steps to extend the expiry date for 10 years:



$ gpg --edit-key [e-mail]
gpg (GnuPG) 1.4.7; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
Secret key is available.
pub  1024D/0E3606D9  created: 2005-09-29  expires: never       usage: SC  
                     trust: unknown       validity: unknown
sub  2048g/AFE99880  created: 2005-09-29  expires: never       usage: E   
sub  1024D/1AF32821  created: 2005-09-29  expired: 2006-09-29  usage: S   
[ unknown] (1). Mozilla Software Releases <[e-mail]>


Command> fpr
pub   1024D/0E3606D9 2005-09-29 Mozilla Software Releases <[e-mail]>
 Primary key fingerprint: F57F 372A 8C6D 4F55 72DE  C9B9 696E 3431 0E36 06D9


Command> key 2
pub  1024D/0E3606D9  created: 2005-09-29  expires: never       usage: SC  
                     trust: unknown       validity: unknown
sub  2048g/AFE99880  created: 2005-09-29  expires: never       usage: E   
sub* 1024D/1AF32821  created: 2005-09-29  expired: 2006-09-29  usage: S   
[ unknown] (1). Mozilla Software Releases <[e-mail]>


Command> expire 
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at Sun Jul  9 17:22:58 2017 PDT
Is this correct? (y/N) y
You need a passphrase to unlock the secret key for
user: "Mozilla Software Releases <[e-mail]>"
1024-bit DSA key, ID 0E3606D9, created 2005-09-29
pub  1024D/0E3606D9  created: 2005-09-29  expires: never       usage: SC  
                     trust: unknown       validity: unknown
sub  2048g/AFE99880  created: 2005-09-29  expires: never       usage: E   
sub* 1024D/1AF32821  created: 2005-09-29  expires: 2017-07-10  usage: S   
[ unknown] (1). Mozilla Software Releases <[e-mail]>


Command> quit
Save changes? (y/N) y
$
$
$ gpg --edit-key [e-mail]
gpg (GnuPG) 1.4.7; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
Secret key is available.
gpg: checking the trustdb
gpg: no ultimately trusted keys found
pub  1024D/0E3606D9  created: 2005-09-29  expires: never       usage: SC  
                     trust: unknown       validity: unknown
sub  2048g/AFE99880  created: 2005-09-29  expires: never       usage: E   
sub  1024D/1AF32821  created: 2005-09-29  expires: 2017-07-10  usage: S   
[ unknown] (1). Mozilla Software Releases <[e-mail]>


Command> fpr
pub   1024D/0E3606D9 2005-09-29 Mozilla Software Releases <[e-mail]>
 Primary key fingerprint: F57F 372A 8C6D 4F55 72DE  C9B9 696E 3431 0E36 06D9


Command> quit
$

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Looks like we have two options here:

1) use "gpg --edit-key" to extend the expiry date of the existing key.

2) generate a completely new key.

Will ask dveditz, window which they prefer.

Comment 5

[Daniel Veditz [:dveditz]]

I guess it depends on how the public keyservers work. Will they notice they need to get an updated copy if the fingerprint matches one they already have?

If you create a new key and upload it to one of them it works its way around to other public key servers.

10 years for a software code-signing key seems awfully long. The point of having an expiration date at all is to put a back-stop on the risk of a compromised key. If a year is too short in practice maybe 18 months or two years instead?

Comment 6

[J. Paul Reed [:preed]]

(In reply to comment #5)

> 10 years for a software code-signing key seems awfully long. The point of
> having an expiration date at all is to put a back-stop on the risk of a
> compromised key. If a year is too short in practice maybe 18 months or two
> years instead?

The issue I'm unclear on is whether we created a detached PGP key for release signing (which is separate and distinct from our releases@ key, as I undersatnd it), or whether they're the same key.

If we can create a new, detached, throw-away key, then we I think we should do that on a yearly basis (we have to do that for our authenticode key anyway). If not, then we should probably think more carefully about how to fix this moving forward.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> I guess it depends on how the public keyservers work. Will they notice they
> need to get an updated copy if the fingerprint matches one they already have?
I've no idea, but will find someone who understands this kinda stuff.


> If you create a new key and upload it to one of them it works its way around to
> other public key servers.
ok.


> 10 years for a software code-signing key seems awfully long. The point of
> having an expiration date at all is to put a back-stop on the risk of a
> compromised key. If a year is too short in practice maybe 18 months or two
> years instead?
I just picked 10years while doing testing. For real use, I'll happily redo this to something lower. Say 2 years? 

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From discussions Thursday/Friday last week with several people, no one is sure exactly if extending certificate expiry date is enough, or whether to just generate a complete new key.

Therefore, in the interest of paranoia, I'll just create full new key.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Now created new public/private key. Basic information is here:

%
% gpg --edit-key [e-mail]
gpg (GnuPG) 1.4.7; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
Secret key is available.
pub  1024D/812347DD  created: 2007-07-17  expires: 2009-07-16  usage: SC  
                     trust: ultimate      validity: ultimate
sub  1024D/17785FE8  created: 2007-07-17  expires: 2009-07-16  usage: S   
sub  2048g/1B0EC2E7  created: 2007-07-17  expires: 2009-07-16  usage: E   
[ultimate] (1). Mozilla Software Releases <[e-mail]>

Command> fpr
pub   1024D/812347DD 2007-07-17 Mozilla Software Releases <[e-mail]>
 Primary key fingerprint: 8D6F 1BA4 A340 4DDB 3F2F  D080 7447 4499 8123 47DD

Command> quit
%
%

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: new public key

This attachment is the new public key. Please contact me offline to see the private key. If these keys look ok to you, please r+.

Comment 11

[Robert Helmer [:rhelmer]]

Comment on attachment 272716 [details] [diff] [review]
new public key

Looks good, but you should add some info at the top like the current one has:

http://stage.mozilla.org/pub/mozilla.org/firefox/releases/1.5/KEY

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: new public key with added header text

Same key as before. However, this has additional manual text info at beginning of file.

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to comment #12)
> Created an attachment (id=272827) [details]
> new public key with added header text
> 
> Same key as before. However, this has additional manual text info at beginning
> of file.

Looks fine, but can you also please sign something and attach the detached .gpg signature file (and tell us what you signed)?

That's the only way I can think of to independently verify that your private key/public key matches and produces what I'd expect.

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Yesterday, I experimented with:
- signing a file, producing external signature file. Using public key to validate external signature.
- encrypting a file, and then decrypting the file.

Today, send email to reviewers with attached files for private review.

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Turns out I incorrectly used "-sa" when signing, which means the generated ascii signature file contains "-----BEGIN PGP MESSAGE-----". However, if I instead use "-clearsign" when signing, the generated ascii signature file contains the expected "-----BEGIN PGP SIGNATURE-----".

Sent new email, with new attachments, to reviewers for private review.

Comment 16

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Reviewer confirmed key ok. However, wanted to re-review when using the exact parameters as specified in the build process. Redone, and sent for re-review.

For reference, the build process uses the following parameters when doing GPG signing:

$ gpg -v -u releases -sba <filename> 

Comment 17

[Daniel Veditz [:dveditz]]

Comment on attachment 272827 [details] [diff] [review]
new public key with added header text

r=dveditz

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

reminder email sent to remaining reviewers.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment on attachment 272827 [details] [diff] [review]
new public key with added header text

after talking with window at coffee machine, now removing window from reviewer list, at her request.

Comment 20

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

All reviews in. Now starting rollout of new key.

Comment 21

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

New public key posted to keymaster.veridis.com. In theory, this should replicate out to other keymaster servers, but its been 6 hours now, and I dont see the new key on other keymaster servers yet. Maybe by morning?


FYI: keymaster.veridis.com lists the old/previous public key as having *no* expiry date?! On wwwkeys.pgp.net, the same old/previous public key has no expiry on the primary key, and an expiry of 2006-09-29 on the secondary key. By comparison, keymaster.veridis.com shows the new public key having the correct expiry date. Not sure what this means about either the old public key or the keymaster servers, but its interesting to note.

Comment 22

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

new key now also visible on wwwkeys.pgp.net and pgp.mit.edu.


Build machines now have new GPG key setup. 

On the build machines, with the new GPG key installed, I tested as follows:
- Copy the most recent release to a temp dir and sign that fake release all over again, following the usual signing process. The usual release process signing instructions worked fine, using the new GPG password. I also tried these same instructions using the old GPG password, and failed (as expected). 
- Verified the signature on the new fake release with the new key. 
- Verified the signature of a previously signed release using the new GPG key, and failed (as expected).

The negative test cases above were to prove that all occurrences of the old expired key have been correctly removed from the build machines, and replaced with the new key.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

This code will need changes
 http://mxr.mozilla.org/mozilla/source/tools/release/Bootstrap/Step/Stage.pm#241
to prevent signing with the new and publishing the old.

Maybe we should just stash the public key somewhere on stage and copy form there each time, rather than the previous release.

Comment 24

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Good catch Nick, thanks for that. 

I had already updated the signing docs, but never dawned on me that the build scripts would do something like this!  Bad assumption on my part.

Reopening bug to track fixing Stage.pm.

Comment 25

[Robert Helmer [:rhelmer]]

(In reply to comment #23)
> This code will need changes
> 
> http://mxr.mozilla.org/mozilla/source/tools/release/Bootstrap/Step/Stage.pm#241
> to prevent signing with the new and publishing the old.
> 
> Maybe we should just stash the public key somewhere on stage and copy form
> there each time, rather than the previous release.


Yeah, I agree.. note the TODO on that line :)

Probably somewhere in /data/cltbld

Comment 26

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I diff'd this KEY file for every Firefox release on stage. 

Every release from 1.5b2 until 2.0.0.6 inclusive was being signed with the same key, now expired. At least this explains why we havent hit this problem before, or at least not in living memory. (Interesting that all releases from 0.10 to 1.5b1 inclusive were signed by Chase's key!)


What about this as a possible fix? 
1) Changing:
my $keyFile = catfile('/home', 'ftp', 'pub', $product, 'releases', '1.5', 'KEY');
...to... 
my $keyFile = catfile('/home', 'ftp', 'pub', $product, 'publickey', 'latest', 'KEY');

2) putting a copy of the latest KEY in that location. putting a copy of the previous keys in there also, for reference. In two years time, when this new key expires, we can put the new key under "latest", and not have to tweak this script again.

Comment 27

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: tweak Stage.pm to use public key from mofo repo, not whatever was found on stage filesystem

Comment 28

[J. Paul Reed [:preed]]

Comment on attachment 279034 [details] [diff] [review]
tweak Stage.pm to use public key from mofo repo, not whatever was found on stage filesystem

>Index: Stage.pm

>+    #my $keyFile = catfile('/home', 'ftp', 'pub', $product, 'releases', '1.5',
>+    #                      'KEY');

Remove this comment, just to keep things tidy.

Otherwise, looks good! r=preed.

Do you need me to land it for you?

Comment 29

[Robert Helmer [:rhelmer]]

Attachment: Stage.pm patch as-landed

Landed this for joduinn:
Checking in Stage.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Stage.pm,v  <--  Stage.pm
new revision: 1.16; previous revision: 1.15
done

Comment 30

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Two changes here:

1) Found all previous Thunderbird and Firefox public keys and checked them in. There were a total of 7 keys checked in, going back to Firefox 0.10. Only 3 keys (including my new one) were actually different keys, the rest were only comment changes. FYI: these keys were all checked in exactly in chronological sequence, so we now have a "history" of our keys.

2) Changed Stage.pm to no longer be hard coded to copy the firefox/releases/1.5/KEY, and instead now get the latest one from CVS. I also grep'd through all the other Bootstrap files to make sure there were no other lurking surprises.

Next time we renew a key, it should be *much* simpler!

Many thanks to Nick for spotting this twist in the plot - otherwise, it would have been a weird surprise during the next release. 

Comment 31

[Robert Helmer [:rhelmer]]

Stage.pm doesn't work with a clean staging area; it's trying to check out into "batch1" directory which hasn't been created yet.

Looks like this directory gets implicitly created further down (as part of creating batch1/prestage):
http://mxr.mozilla.org/mozilla/source/tools/release/Bootstrap/Step/Stage.pm#255

Comment 32

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: patch to create batch1 dir if doesnt already exist

The already-landed patch above works fine in an end-to-end run.

However, if you run *just* the Stage step, it fails out because the batch1 directory does not yet exist. This patch checks for the batch1 directory and creates it if it does not already exist.

Also, by creating a "$batch1Dir" variable, cleaned up a few places where we kept reconstructing the batch1 dirname...

Comment 33

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: patch to create batch1 dir if it doesnt already exist

same as last time, only displays better because I used the correct "cvs diff -u" params.

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 279137 [details] [diff] [review]
patch to create batch1 dir if it doesnt already exist

Looks good. There are a bunch of other places you could use $batch1Dir, eg
  dir => catfile($stageDir, 'batch1', 'key-checkout'),
could become
  dir => catfile($batch1Dir, 'key-checkout'),

I don't have a strong opinion about that.

Comment 35

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

cf landed this patch for me.

Comment 36

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

reopening after FF2.0.0.7 because of the following questions:

1) Should the old expired key be revoked from key servers? 
There are people using older releases who would need to see this older expired key, so we think we should not revoke the older expired key. 

2) Should the new key be signed by the old key?
This would allow users to better trust the new key. This seems reasonable, and we are doing this.

Comment 37

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #36)
> reopening after FF2.0.0.7 because of the following questions:
> 
> 1) Should the old expired key be revoked from key servers? 
> There are people using older releases who would need to see this older expired
> key, so we think we should not revoke the older expired key. 
> 
> 2) Should the new key be signed by the old key?
> This would allow users to better trust the new key. This seems reasonable, and
> we are doing this.
From triage, unclear of value of doing this at this point. We've been using this "new" key since aug2008.