1371318 - Generate certificate to sign Focus APK

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

Focus for Android will soon be shipped to Google Play. Here's what we need to do:

Before the release day:
1. SSH to a Linux (android) signing server
2. Manually Create a new keystore dedicated to host the new cert[1]. For future automation, passwords for both keystore and certificate must be the same
3. Sign the APK[2]
4. Give the APK back to the Focus team.

Right after the first APK is signed:
I. Store the passwords on the private repo.
II. Copy the keystore to the other signing servers. Replication is usually done by archiving the keystore in a zip-encrypted archive. The passphrase is usually 80 chars long
III. Create two offline backups of the keys[3].

For next releases:
A. On the signing server, create a new signing format, called something like "focus-jar"
B. Create new instances of the signing server dedicated to that new format. This can be done by either a new process on the same machine (but a different port) or by creating a new machine.
C. See whether the Focus team is ready to port a part of their automation to Taskcluster. This would ensure the security of the signing process.



[1] https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-Jarsigning(APK)
[2] https://developer.android.com/studio/publish/app-signing.html#sign-apk
[3] https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-Backups

Comment 1

[Aki Sasaki (not active)]

Thanks Johan!

(In reply to Johan Lorenzo [:jlorenzo] from comment #0)
> Focus for Android will soon be shipped to Google Play. Here's what we need
> to do:
> 
> Before the release day:
> 1. SSH to a Linux (android) signing server
> 2. Manually Create a new keystore dedicated to host the new cert[1]. For
> future automation, passwords for both keystore and certificate must be the
> same

We can do step 2 beforehand, if time is an issue.

> 3. Sign the APK[2]
> 4. Give the APK back to the Focus team.
> 
> Right after the first APK is signed:
> I. Store the passwords on the private repo.
> II. Copy the keystore to the other signing servers. Replication is usually
> done by archiving the keystore in a zip-encrypted archive. The passphrase is
> usually 80 chars long

I usually use `pwgen 80` for the passphrase, scp down to my laptop, scp up to the target server, unzip.
secure wipe (srm, rm -P, etc) the files on laptop afterwards.

> III. Create two offline backups of the keys[3].
> 
> For next releases:
> A. On the signing server, create a new signing format, called something like
> "focus-jar"

If we have a new instance in (B), we can reuse the 'jar' format.

> B. Create new instances of the signing server dedicated to that new format.
> This can be done by either a new process on the same machine (but a
> different port) or by creating a new machine.
> C. See whether the Focus team is ready to port a part of their automation to
> Taskcluster. This would ensure the security of the signing process.
> 
> 
> 
> [1]
> https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-Jarsigning(APK)
> [2] https://developer.android.com/studio/publish/app-signing.html#sign-apk
> [3] https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-Backups

Comment 2

[Johan Lorenzo [:jlorenzo]]

Thank you for the explanations, Aki! I forgot to call out who helped me list out the steps. Thank you :bhearsum, as well.


> 2. Manually Create a new keystore dedicated to host the new cert[1]. For future automation, passwords for both keystore and certificate must be the same

I created a keystore called "focus-jar" in /builds/signing/rel-key-signing-server/secrets. The certificate in there has the alias "focus".

Just like the release certificate, I filled these values:
> CN=Release Engineering, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US

I also needed to chmod 600 the keystore file.


> 3. Sign the APK[2]
Done. There are actually 2 APKs: one Focus, the other for the german-speaking population: Klar. I did it with the command `jarsigner -keystore /builds/signing/rel-key-signing-server/secrets/focus-jar app-focus-webkit-release-unsigned.apk focus`

I checked the signatures with `jarsigner -verify -verbose -keystore /builds/signing/rel-key-signing-server/secrets/focus-jar app-focus-webkit-release.apk focus`

I also needed to zipalign the APK one more time (the unsigned version was already zipaligned). I did it on my local machine with `/opt/android-sdk/build-tools/25.0.3/zipalign -v 4 app-klar-webkit-release.apk app-klar-webkit-release-aligned.apk `


> 4. Give the APK back to the Focus team.
Before going further, would you mind double checking the signatures, Aki? I left the APKs at: https://drive.google.com/drive/folders/0B7Y2LpZbqKlucjNPQ2lIcktNcG8?usp=sharing


> I. Store the passwords on the private repo.
The keystore/cert password has been pushed to the private repo as a separate file. Aki, could you move it to the correct place? 


> II. Copy the keystore to the other signing servers
Copies now exist on signing{4,5,6}. I verified the signatures against the new keystores. I also listed the certs present in the new keystores, in order to double-check the keystore pass: `keytool -list -keystore /builds/signing/rel-key-signing-server/secrets/focus-jar -alias focus`

I shredded the temporary zip on every machine: `shred --iterations=7 --remove focus.zip`.

Comment 3

[Johan Lorenzo [:jlorenzo]]

While Aki reviews the signatures, I opened the folder where signed APKs are[1] to Sylvestre and Sebastian.

[1] https://drive.google.com/drive/folders/0B7Y2LpZbqKlucjNPQ2lIcktNcG8

Comment 4

[Aki Sasaki (not active)]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)
> > 4. Give the APK back to the Focus team.
> Before going further, would you mind double checking the signatures, Aki? I
> left the APKs at:
> https://drive.google.com/drive/folders/
> 0B7Y2LpZbqKlucjNPQ2lIcktNcG8?usp=sharing

Looks like they're both signed with

Owner: CN=Release Engineering, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US
Issuer: CN=Release Engineering, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US
Serial number: 593a6dd3
Valid from: Fri Jun 09 02:43:47 PDT 2017 until: Tue Oct 25 02:43:47 PDT 2044
Certificate fingerprints:
         MD5:  53:9F:E6:6E:3B:04:69:55:3F:0C:E2:84:CC:F9:01:F8
         SHA1: 5E:F5:AE:40:28:C9:84:92:E2:B2:AD:E3:4F:FF:28:66:05:D5:06:8F
         SHA256: 62:03:A4:73:BE:36:D6:4E:E3:7F:87:FA:50:0E:DB:C7:9E:AB:93:06:10:AB:9B:9F:A4:CA:7D:5C:1F:1B:4F:FC
         Signature algorithm name: SHA256withRSA
         Version: 3

> > I. Store the passwords on the private repo.
> The keystore/cert password has been pushed to the private repo as a separate
> file. Aki, could you move it to the correct place? 

a) I don't see any new commit from you.  Are you sure you added it?  The .gitignore is intentially strict.
b) This probably belongs in a new signing-server file, probably signing-server-focus.txt.gpg, so we are able to grant people access to that without the rest of the release keys.  This is less important while the key lives on the release signing server instances, but would allow us to spin up a new instance with a separate access list later.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Aki, does that mean it is ok or not?
I would like to upload the apk during the week end

Comment 6

[Aki Sasaki (not active)]

lgtm.  :jlorenzo will need to add the passphrase to the secrets repo for future use.

Comment 7

[Johan Lorenzo [:jlorenzo]]

> a) I don't see any new commit from you.  Are you sure you added it?  The .gitignore is intentially strict.
Sorry about this, I actually forgot to push my commit *facepalm*. This is now in the private repo.

> b) This probably belongs in a new signing-server file, probably signing-server-focus.txt.gpg
Agreed and done.

Comment 8

[Johan Lorenzo [:jlorenzo]]

New 1.0 candidates APKs have to be signed. I'm on it.

Comment 9

[Johan Lorenzo [:jlorenzo]]

Another set of candidates came in.

Comment 10

[Johan Lorenzo [:jlorenzo]]

The candidate branded RC3 just arrived. Signing these ones.

Comment 11

[Johan Lorenzo [:jlorenzo]]

:hwine reached out to me about offline backups. I'm going to follow the procedure at https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-Backups. I'll transfer the backup to :hwine today at around 9:00am PDT.

Comment 12

[Johan Lorenzo [:jlorenzo]]

Transfer done. Hal called out a small issue: Chris Cooper is still listed on the mana page, but he's not a part of the releng group anymore. As a consequence, he should not be on the recipient list of the encrypted archive.

Because I don't have the decrypted files on my computer I couldn't quickly regenerate the archive. Hal and I agreed on leaveing the re-encryption to one of the recipient. In fact, we do need someone to double-check I didn't omit an information in the archive.

Hal suggested either Rail or Aki to perform this.

Comment 13

[Johan Lorenzo [:jlorenzo]]

RC4 came up. I'm going to sign them.

Comment 14

[Hal Wine [:hwine] use NI!]

(In reply to Johan Lorenzo [:jlorenzo] from comment #12)
> Transfer done. Hal called out a small issue: Chris Cooper is still listed on
> the mana page, but he's not a part of the releng group anymore. As a
> consequence, he should not be on the recipient list of the encrypted archive.

Aki removed coop before the tarball was placed in the safe. \o/

Comment 15

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483

Comment 16

[Justin Wood (:Callek)]

found in triage, is this done?

Comment 17

[Johan Lorenzo [:jlorenzo]]

I confirm this was done in spring 2017. Since then, we've moved the private key to Autograph (bug 1492245). I don't think this bug should be private anymore (like bug 1492245).

Thanks for finding it, Callek!