sign Fennec and Focus with autograph

VERIFIED FIXED in Firefox -esr60

Status

defect
VERIFIED FIXED
8 months ago
2 months ago

People

(Reporter: g-k, Assigned: jlorenzo)

Tracking

(Blocks 1 bug, {leave-open})

unspecified
Firefox 65
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 fixed, firefox65 affected)

Details

Attachments

(12 attachments, 2 obsolete attachments)

55 bytes, text/x-github-pull-request
mtabara
: review+
Details | Review
57 bytes, text/x-github-pull-request
Details | Review
897 bytes, patch
Details | Diff | Splinter Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
57 bytes, text/x-github-pull-request
Details | Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
47 bytes, text/x-phabricator-request
mtabara
: review+
Details | Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
55 bytes, text/x-github-pull-request
jlorenzo
: review+
Details | Review
55 bytes, text/x-github-pull-request
mtabara
: review+
Details | Review
(Reporter)

Description

8 months ago
As discussed in email (subject: "APK signing for Fennec and Focus/Klar with autograph").

We can sign Fennec and Focus/Klar with TC and signingscript like we do for Fenix (assuming the code lives in central).

Steps:

1. :jlorenzo send gpg-encrypted Focus/Klar APK keys to :g-k
2. :g-k to:
  a) update config with creds for Fenix and Focus to dummy APK signer in stage and real keys in prod
  b) redeploy autograph with new keys
3. :jlorenzo does TC/signingscript magic to verify signing works
(Reporter)

Updated

8 months ago
Component: Security → Autograph
QA Contact: jvehent → gguthe
(Assignee)

Updated

7 months ago
Duplicate of this bug: 1497474
Johan is ramping-me up in the mobile/signing world so I'll help with this handover tomorrow.

For posterity, the following step:

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #0)
> As discussed in email (subject: "APK signing for Fennec and Focus/Klar with
> autograph").
> 1. :jlorenzo send gpg-encrypted Focus/Klar APK keys to :g-k

becomes:

1. grab the keys (which are passphrase protected already IIUC)
2.  zip-encrypt with 80-char-long passwords 
3. magic-wormhole the zip archives to SecOps
4. paste the password via Signal to SecOps

All Signal-messages are short-time lived.

(even though the key is passphrase protected, we still zip-encypt as it doesn't hurt to have an extra-layer)
Thank you Greg and Johan for the help with transferring this today.
We've successfully sent all the keys to Greg (dep, nightly,release keys) + the focus jar one. 

Next steps:
* Greg to update the Autograph configs and send us the credentials
* RelEng updates hiera and preps the PR
* roll-out testing stuff gradually.
FWIW - I've added a docs-comment here[1] for posterity in case we need to redo something similar. 

[1]: https://mana.mozilla.org/wiki/display/RelEng/Signing?focusedCommentId=85656138#comment-85656138
(Reporter)

Comment 5

7 months ago
Got the keys, updated the configs, and tested all but one against autograph locally. Sent the autograph creds to :mtabara and :jlorenzo.

Still need to redeploy with the updated configs and debug the last key.
(Reporter)

Comment 6

7 months ago
OK to unblock testing I redeployed to stage and prod (in the separate autograph/HSM env we use for TC and signingscript) with the last key disabled.
(Assignee)

Comment 7

7 months ago
Assignee: nobody → jlorenzo
Attachment #9017485 - Flags: review?(mtabara)
(Assignee)

Updated

7 months ago
Assignee: jlorenzo → mtabara
(Reporter)

Updated

7 months ago
See Also: → 1499351
Attachment #9017485 - Flags: review?(mtabara) → review+
(Assignee)

Comment 11

7 months ago
Comment on attachment 9017826 [details] [review]
[puppet] Switch Fennec dep key to Autograph

https://github.com/mozilla-releng/build-puppet/pull/269#pullrequestreview-165635542
Attachment #9017826 - Flags: review?(jlorenzo) → review+
Status so far: tested via three methods the validity of the APK signing.

1. 
`jarsigner -verify -certs` shows both of them properly signed

2. 

`keytool -printcert -file autograph_target/META-INF/SIGNATURE.RSA`
`keytool -printcert -file jar_target/META-INF/DEP.RSA`

return the same results.

3. 
`jarsigner -verify -strict -verbose -keystore temp target.apk.1 dep` shows them correctly signed.
(`temp` being a keystore and `dep` being an alias)

Issues:
* we switched from sha1 to sha256 which will be a proble for existing users of older android phones.
* (irrelevant IIUC) the META_INF filesa in autograph jar are named “SIGNATURE” instead of formerly known “DEP” (both the RSA and SF)

We did this before and failed like in bug 1332916. In order to fix that, a sanity check was added in pushapkscript[1] so that only sha1 to be allowed. If we shipped with a different sha algorithm, people with old phones won't be able to install Firefox anymore.

So the solution here is to provide sha1 support in Autograph. This was tracked here[2] and landed AFAIK so there must be some config issue missing somewhere. Or it landed and I've been using Fennec jar from a while ago.

Note to self: 
* talk to SecOps on Monday morning to see if there's anything that needs to be amended in the autograph_fennec format client to default to sha1 instead of sha256.
* once this is working, redo the Fennec staging release and make sure sha is correct
* redo the three validty tests aforementioned 
* land the maple patch in-tree on central and start prepping beta counterpart and plan with QA for 64.0b9

[1]: https://github.com/mozilla-releng/pushapkscript/pull/13/files
[2]: https://github.com/mozilla-services/autograph/issues/156
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13)
> Note to self: 
> * talk to SecOps on Monday morning to see if there's anything that needs to
> be amended in the autograph_fennec format client to default to sha1 instead
> of sha256.
> * once this is working, redo the Fennec staging release and make sure sha is
> correct
> * redo the three validty tests aforementioned 
> * land the maple patch in-tree on central and start prepping beta
> counterpart and plan with QA for 64.0b9


19:12:07 <~ulfr> ok, so apk signing was pinned to sha256 until I merged the dsa support today
19:12:07 <mtabara> w00t!
19:12:09 <~ulfr> we now have an option to request sha1
19:12:17 <~ulfr> it's not deployed yet
19:12:46 <~ulfr> https://github.com/mozilla-services/autograph/pull/166/files
19:16:16 <~ulfr> mtabara: if you could add that option to the fennec job, that'll help rollout 
19:16:34 <~ulfr> it can be added now and autograph will ignore it until the new version is deployed
19:21:24 <mtabara> ulfr: that's great! yeah, I'll do that. I need to dig though as I don't know how that translates for the task. I'd assume I need to add a new format scope which signingscript will translate into that additional argument you mentioned
19:22:25 <~ulfr> jlorenzo's patch from earlier also adds an option to that same call, it's almost identical
Just noticed I'm assigned to the bug but Johan and myself are splitting the work so unassigning myself in the rule of fairness ;-)
Assignee: mtabara → nobody
Status update on Fennec side:
* I've changed the scopes on maple to point Fennec signing jobs to `autograph_fennec_sha1`
* 9.4.1 signingscript is rolled-out to send that additional pkdigest set to SHA1 for `autograph_fennec_sha1` scopes
* puppet is updated to reflect in its dep-passwords the autograph configs / scopes for this particular case

2.4.0 landed last week doesn't encompass the support for pk7 digest. But likely the 2.5 will. See[1], upcoming this week.
In 162 PR[2] there are two changes: 
a) support for DSA key
b) sha1 digest support

Idealy both work and we can be unblocked on all fronts for all channels. 
If DSA support won't work properly, that's going to be commented out and allow at least the sha1 support rolled-out so that we can test the dep and release keys

[1]: https://github.com/mozilla-services/autograph/compare/2.4.0...2.5.0
[2]: https://github.com/mozilla-services/autograph/pull/162/files
(Assignee)

Comment 19

6 months ago
Comment on attachment 9024005 [details] [review]
[signingscript] enable custom digest in APK for Autograph

Was r+'d at https://github.com/mozilla-releng/signingscript/pull/89#pullrequestreview-173897032
Attachment #9024005 - Flags: review?(jlorenzo) → review+
(Assignee)

Comment 20

6 months ago
Comment on attachment 9024393 [details] [review]
[puppet] Switch Fennec dep key to SHA1 signing format

Was r+'d at https://github.com/mozilla-releng/build-puppet/pull/294#pullrequestreview-173898324
Attachment #9024393 - Flags: review?(jlorenzo) → review+
(Assignee)

Comment 21

6 months ago
Assignee: nobody → jlorenzo
(Assignee)

Comment 23

6 months ago
Switch Fennec format from `jar` to `autograph_apk_fennec_sha1`
Status: NEW → RESOLVED
Last Resolved: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 65
(Assignee)

Updated

6 months ago
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Nightlies triggered encompassing this change look good on signing side. E.g. build-signing task[1] but failed on publishing the APK. That's good since pushapk caught the error \o/. Let's see what jarsigner[2] is complaining[3] for. 

[1]: https://tools.taskcluster.net/groups/Ky4MQEajSNeFMeQI51enVA/tasks/N_KSKOCHRIGDezNyLaPqSA/details
[2]: https://tools.taskcluster.net/groups/Ky4MQEajSNeFMeQI51enVA/tasks/HbAQzMDjRm6gfzfgtEVEyQ/details
[3]: https://taskcluster-artifacts.net/HbAQzMDjRm6gfzfgtEVEyQ/0/public/logs/live_backing.log
(Assignee)

Comment 30

6 months ago
I don't think this bug should be private anymore. It doesn't contain neither any security-sensitive information, nor any private keys.

:bc, :WG9s[1], per bug 1332916, you have some devices that are old enough. We just changed the way Nightly is signed. Now, APKs are signed with 2 digest algorithm: SHA1 and SHA-256. From a specs perpective, this shouldn't break devices that require SHA1. I don't have such device, though. Are you guys able to get the latest version from the Google Play store? If not you can manually download [2]. Please let me know how this goes for you.

[1] I couldn't NI you, because the account is apparently disabled.
[2] https://queue.taskcluster.net/v1/task/RipvdTftTgCWlInECTwnCg/runs/1/artifacts/public/build/target.apk (arm) or https://queue.taskcluster.net/v1/task/FZW49Xl8SAGkXcEhnzR5TA/runs/1/artifacts/public/build/target.apk (x86)
Group: mozilla-employee-confidential
Depends on: 1332916
Flags: needinfo?(bob)

Comment 32

6 months ago
Autophone is no more and the lowest Android version we are testing on hardware is 7.0. We still test Android 4.2/4.3 emulators though. You can start an emulator from a build environment via ./mach android-emulator --version '4.3'
Flags: needinfo?(bob)
Comment hidden (Intermittent Failures Robot)
(Assignee)

Updated

6 months ago
Duplicate of this bug: 1509473
(Assignee)

Comment 35

6 months ago
Thank you for the information, Bob! I installed [1] on an Emulator, and it ran. I'm also following the update trends on Google Play (for Android 4.x). Numbers are still lower than usual, but last week's lowest point was last Thursday (US Thanksgiving). 

[1] https://archive.mozilla.org/pub/mobile/nightly/2018/11/2018-11-26-10-00-51-mozilla-central-android-api-16/fennec-65.0a1.multi.android-arm.apk
(Assignee)

Updated

5 months ago
Blocks: 1513013
(Assignee)

Comment 36

5 months ago
Focus nightly and release have been migrated without a single issue. Fennec nightly too. Fennec beta hit bug 1513564. I'm then going to close this bug in favor of bug 1513564.
Blocks: 1513564
Status: REOPENED → RESOLVED
Last Resolved: 6 months ago5 months ago
Resolution: --- → FIXED
(Assignee)

Comment 37

5 months ago
Attachment #9032177 - Flags: review?(mtabara)
Comment on attachment 9032177 [details] [review]
[build/puppet] Clean up Focus passwords

Nits in PR.
Attachment #9032177 - Flags: review?(mtabara) → review+
(Assignee)

Updated

5 months ago
Depends on: 1371318

(In reply to Johan Lorenzo [:jlorenzo] from comment #36)

Focus nightly and release have been migrated without a single issue. Fennec
nightly too. Fennec beta hit bug 1513564. I'm then going to close this bug
in favor of bug 1513564.

To confirm: Fennec release went on smoothly earlier today with Autograph signing builds (e.g. https://tools.taskcluster.net/groups/HdMjfb7ZQbuNGnTdDtQxjg/tasks/CNM-E1wTQEa5UlWs26ryMw/details).

Status: RESOLVED → VERIFIED
(Assignee)

Updated

2 months ago
Blocks: 1535419
You need to log in before you can comment on or make changes to this bug.