1492245 - sign Fennec and Focus with autograph

Status: VERIFIED FIXED

(Cloud Services :: Operations: Autograph, defect)

Description

[Greg G]

As discussed in email (subject: "APK signing for Fennec and Focus/Klar with autograph").

We can sign Fennec and Focus/Klar with TC and signingscript like we do for Fenix (assuming the code lives in central).

Steps:

1. :jlorenzo send gpg-encrypted Focus/Klar APK keys to :g-k
2. :g-k to:
  a) update config with creds for Fenix and Focus to dummy APK signer in stage and real keys in prod
  b) redeploy autograph with new keys
3. :jlorenzo does TC/signingscript magic to verify signing works

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Johan is ramping-me up in the mobile/signing world so I'll help with this handover tomorrow.

For posterity, the following step:

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #0)
> As discussed in email (subject: "APK signing for Fennec and Focus/Klar with
> autograph").
> 1. :jlorenzo send gpg-encrypted Focus/Klar APK keys to :g-k

becomes:

1. grab the keys (which are passphrase protected already IIUC)
2.  zip-encrypt with 80-char-long passwords 
3. magic-wormhole the zip archives to SecOps
4. paste the password via Signal to SecOps

All Signal-messages are short-time lived.

(even though the key is passphrase protected, we still zip-encypt as it doesn't hurt to have an extra-layer)

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Thank you Greg and Johan for the help with transferring this today.
We've successfully sent all the keys to Greg (dep, nightly,release keys) + the focus jar one. 

Next steps:
* Greg to update the Autograph configs and send us the credentials
* RelEng updates hiera and preps the PR
* roll-out testing stuff gradually.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

FWIW - I've added a docs-comment here[1] for posterity in case we need to redo something similar. 

[1]: https://mana.mozilla.org/wiki/display/RelEng/Signing?focusedCommentId=85656138#comment-85656138

Comment 5

[Greg G]

Got the keys, updated the configs, and tested all but one against autograph locally. Sent the autograph creds to :mtabara and :jlorenzo.

Still need to redeploy with the updated configs and debug the last key.

Comment 6

[Greg G]

OK to unblock testing I redeployed to stage and prod (in the separate autograph/HSM env we use for TC and signingscript) with the last key disabled.

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: [signingscript] PR: part 1: Swap Focus signing to autograph

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: [focus-android] PR: Swap Focus signing to autograph

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [puppet] Switch Fennec dep key to Autograph

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [in-tree] Switch Fennec format from `jar` to `autograph_fennec`

Comment 11

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9017826 [details] [review]
[puppet] Switch Fennec dep key to Autograph

https://github.com/mozilla-releng/build-puppet/pull/269#pullrequestreview-165635542

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9017826 [details] [review]
[puppet] Switch Fennec dep key to Autograph

Landed in production: https://github.com/mozilla-releng/build-puppet/commit/3b552b3c212c024424f6b9ab29341efa70356658

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Status so far: tested via three methods the validity of the APK signing.

1. 
`jarsigner -verify -certs` shows both of them properly signed

2. 

`keytool -printcert -file autograph_target/META-INF/SIGNATURE.RSA`
`keytool -printcert -file jar_target/META-INF/DEP.RSA`

return the same results.

3. 
`jarsigner -verify -strict -verbose -keystore temp target.apk.1 dep` shows them correctly signed.
(`temp` being a keystore and `dep` being an alias)

Issues:
* we switched from sha1 to sha256 which will be a proble for existing users of older android phones.
* (irrelevant IIUC) the META_INF filesa in autograph jar are named “SIGNATURE” instead of formerly known “DEP” (both the RSA and SF)

We did this before and failed like in bug 1332916. In order to fix that, a sanity check was added in pushapkscript[1] so that only sha1 to be allowed. If we shipped with a different sha algorithm, people with old phones won't be able to install Firefox anymore.

So the solution here is to provide sha1 support in Autograph. This was tracked here[2] and landed AFAIK so there must be some config issue missing somewhere. Or it landed and I've been using Fennec jar from a while ago.

Note to self: 
* talk to SecOps on Monday morning to see if there's anything that needs to be amended in the autograph_fennec format client to default to sha1 instead of sha256.
* once this is working, redo the Fennec staging release and make sure sha is correct
* redo the three validty tests aforementioned 
* land the maple patch in-tree on central and start prepping beta counterpart and plan with QA for 64.0b9

[1]: https://github.com/mozilla-releng/pushapkscript/pull/13/files
[2]: https://github.com/mozilla-services/autograph/issues/156

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13)
> Note to self: 
> * talk to SecOps on Monday morning to see if there's anything that needs to
> be amended in the autograph_fennec format client to default to sha1 instead
> of sha256.
> * once this is working, redo the Fennec staging release and make sure sha is
> correct
> * redo the three validty tests aforementioned 
> * land the maple patch in-tree on central and start prepping beta
> counterpart and plan with QA for 64.0b9


19:12:07 <~ulfr> ok, so apk signing was pinned to sha256 until I merged the dsa support today
19:12:07 <mtabara> w00t!
19:12:09 <~ulfr> we now have an option to request sha1
19:12:17 <~ulfr> it's not deployed yet
19:12:46 <~ulfr> https://github.com/mozilla-services/autograph/pull/166/files
19:16:16 <~ulfr> mtabara: if you could add that option to the fennec job, that'll help rollout 
19:16:34 <~ulfr> it can be added now and autograph will ignore it until the new version is deployed
19:21:24 <mtabara> ulfr: that's great! yeah, I'll do that. I need to dig though as I don't know how that translates for the task. I'd assume I need to add a new format scope which signingscript will translate into that additional argument you mentioned
19:22:25 <~ulfr> jlorenzo's patch from earlier also adds an option to that same call, it's almost identical

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Just noticed I'm assigned to the bug but Johan and myself are splitting the work so unassigning myself in the rule of fairness ;-)

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [signingscript] enable custom digest in APK for Autograph

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [puppet] Switch Fennec dep key to SHA1 signing format

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Status update on Fennec side:
* I've changed the scopes on maple to point Fennec signing jobs to `autograph_fennec_sha1`
* 9.4.1 signingscript is rolled-out to send that additional pkdigest set to SHA1 for `autograph_fennec_sha1` scopes
* puppet is updated to reflect in its dep-passwords the autograph configs / scopes for this particular case

2.4.0 landed last week doesn't encompass the support for pk7 digest. But likely the 2.5 will. See[1], upcoming this week.
In 162 PR[2] there are two changes: 
a) support for DSA key
b) sha1 digest support

Idealy both work and we can be unblocked on all fronts for all channels. 
If DSA support won't work properly, that's going to be commented out and allow at least the sha1 support rolled-out so that we can test the dep and release keys

[1]: https://github.com/mozilla-services/autograph/compare/2.4.0...2.5.0
[2]: https://github.com/mozilla-services/autograph/pull/162/files

Comment 19

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9024005 [details] [review]
[signingscript] enable custom digest in APK for Autograph

Was r+'d at https://github.com/mozilla-releng/signingscript/pull/89#pullrequestreview-173897032

Comment 20

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9024393 [details] [review]
[puppet] Switch Fennec dep key to SHA1 signing format

Was r+'d at https://github.com/mozilla-releng/build-puppet/pull/294#pullrequestreview-173898324

Comment 21

[Johan Lorenzo [:jlorenzo]]

Attachment: [focus-android] PR: Swap Focus **release** signing to autograph

Comment 22

[Johan Lorenzo [:jlorenzo]]

Attachment: [puppet] Expose Fennec on Autograph

r+'d by Mihai at https://github.com/mozilla-releng/build-puppet/pull/306#pullrequestreview-177697996
Landed on master at https://github.com/mozilla-releng/build-puppet/commit/b820c2ace1a3eb608330188e54b5a6d4b294ef8d

Comment 23

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1492245 - Switch Fennec format from `jar` to `autograph_apk_fennec_sha1` r=mtabara

Switch Fennec format from `jar` to `autograph_apk_fennec_sha1`

Comment 24

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/275ccc8b8fe7

Comment 25

[Mihai Tabara [:mtabara]⌚️GMT]

Nightlies triggered encompassing this change look good on signing side. E.g. build-signing task[1] but failed on publishing the APK. That's good since pushapk caught the error \o/. Let's see what jarsigner[2] is complaining[3] for. 

[1]: https://tools.taskcluster.net/groups/Ky4MQEajSNeFMeQI51enVA/tasks/N_KSKOCHRIGDezNyLaPqSA/details
[2]: https://tools.taskcluster.net/groups/Ky4MQEajSNeFMeQI51enVA/tasks/HbAQzMDjRm6gfzfgtEVEyQ/details
[3]: https://taskcluster-artifacts.net/HbAQzMDjRm6gfzfgtEVEyQ/0/public/logs/live_backing.log

Comment 26

[Johan Lorenzo [:jlorenzo]]

Attachment: [signingscript] Generalize APK format to `autograph_apk_*`

r+'d by mtabara at https://github.com/mozilla-releng/signingscript/pull/93#pullrequestreview-177691932

Comment 27

[Johan Lorenzo [:jlorenzo]]

Attachment: [signingscript] Fix SHA1 detection for APKs

r+'d by mtabara at https://github.com/mozilla-releng/signingscript/pull/95/#pullrequestreview-177920493

Comment 28

[Johan Lorenzo [:jlorenzo]]

Attachment: [pushapkscript] Parse manifest to detect what digest algorithm was used

r+'d by mtabara https://github.com/mozilla-releng/pushapkscript/pull/52#pullrequestreview-177963554

Comment 29

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Bug 1492245 - Bump signingscript to 9.5.1

r+'d by mtabara at https://github.com/mozilla-releng/build-puppet/pull/307#pullrequestreview-177927414

Comment 30

[Johan Lorenzo [:jlorenzo]]

I don't think this bug should be private anymore. It doesn't contain neither any security-sensitive information, nor any private keys.

:bc, :WG9s[1], per bug 1332916, you have some devices that are old enough. We just changed the way Nightly is signed. Now, APKs are signed with 2 digest algorithm: SHA1 and SHA-256. From a specs perpective, this shouldn't break devices that require SHA1. I don't have such device, though. Are you guys able to get the latest version from the Google Play store? If not you can manually download [2]. Please let me know how this goes for you.

[1] I couldn't NI you, because the account is apparently disabled.
[2] https://queue.taskcluster.net/v1/task/RipvdTftTgCWlInECTwnCg/runs/1/artifacts/public/build/target.apk (arm) or https://queue.taskcluster.net/v1/task/FZW49Xl8SAGkXcEhnzR5TA/runs/1/artifacts/public/build/target.apk (x86)

Comment 31

[Johan Lorenzo [:jlorenzo]]

The dep key is also working fine, per pushapk: https://tools.taskcluster.net/groups/MNtp86r_TD6GspWmUzYCVA/tasks/F836a8AFThima3DBpkLzRg/runs/1/logs/public%2Flogs%2Flive_backing.log#L7

Comment 32

[Bob Clary [:bc] (inactive)]

Autophone is no more and the lowest Android version we are testing on hardware is 7.0. We still test Android 4.2/4.3 emulators though. You can start an emulator from a build environment via ./mach android-emulator --version '4.3'

Comment 33

[Intermittent Failures Robot]

1 failures in 2387 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* Android: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1492245&startday=2018-11-19&endday=2018-11-25&tree=all

Comment 35

[Johan Lorenzo [:jlorenzo]]

Thank you for the information, Bob! I installed [1] on an Emulator, and it ran. I'm also following the update trends on Google Play (for Android 4.x). Numbers are still lower than usual, but last week's lowest point was last Thursday (US Thanksgiving). 

[1] https://archive.mozilla.org/pub/mobile/nightly/2018/11/2018-11-26-10-00-51-mozilla-central-android-api-16/fennec-65.0a1.multi.android-arm.apk

Comment 36

[Johan Lorenzo [:jlorenzo]]

Focus nightly and release have been migrated without a single issue. Fennec nightly too. Fennec beta hit bug 1513564. I'm then going to close this bug in favor of bug 1513564.

Comment 37

[Johan Lorenzo [:jlorenzo]]

Attachment: [build/puppet] Clean up Focus passwords

Comment 38

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9032177 [details] [review]
[build/puppet] Clean up Focus passwords

Nits in PR.

Comment 39

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/ed897e983193

Comment 40

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Johan Lorenzo [:jlorenzo] from comment #36)

Focus nightly and release have been migrated without a single issue. Fennec
nightly too. Fennec beta hit bug 1513564. I'm then going to close this bug
in favor of bug 1513564.

To confirm: Fennec release went on smoothly earlier today with Autograph signing builds (e.g. https://tools.taskcluster.net/groups/HdMjfb7ZQbuNGnTdDtQxjg/tasks/CNM-E1wTQEa5UlWs26ryMw/details).

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d6a7bf9bfa5f
[focus] Bug 1492245 - part 1: Sign nighlies with autograph
https://hg.mozilla.org/integration/autoland/rev/9d0651447fcb
[focus] Bug 1492245 - part 2: Sign releases with autograph