1842229 - Please create new signer IDs to enable APK v3 signatures

Status: RESOLVED DUPLICATE of bug 1703293

(Cloud Services :: Operations: Autograph, task)

Description

[Johan Lorenzo [:jlorenzo]]

autograph#645 added the ability to sign APKs with v3 signatures. To my knowledge, we've never used this feature before. While looking at bug 1838680, I realized the implementation seems low risk and I think we can quickly get Fenix Nightly signed with v3 signatures.

To do so, I'd recommend creating new signer IDs by copying existing ones and adding:

mode: v3enabled

This way, we can roll out v3 signatures by riding the trains. Per RELENG-718, there are 4 products we should enable: Fenix, Focus/Klar, Reference-Browser, and Mozilla VPN. I know Fenix has 3 keys (nightly, beta, and releases). All projects should have dep keys too. So we're looking at something like a dozens of signers to create.

How does that sound to you, :hwine?

[1] https://source.android.com/docs/security/features/apksigning/v3

Comment 1

[hwine]

Two questions:

1. deployment approach

   To do so, I'd recommend creating new signer IDs by copying existing ones and adding:

   If I understand things correctly, your current signers are capable of performing either v1 or v2 signing. I.e you code is already setting a mode when doing APK signing.

   Given that, is there a preference on your end to have a separate signer for v3 (which would also be capable of v1 & v2 signing - you'd only get v3 if explicitly requested). If not, I'd prefer to add the v3 enablement to an existing signer, to reduce configuration complexity.

2. use of stage autograph
   Also, could you clarify if you have the ability to test any of this in stage. That would greatly speed up the iteration process (if any is needed). I'd be happy to add signers to stage if needed. We were rushed the last time and elected to take the risk of test-in-prod, but getting a way to test in stage would be much preferred.

Comment 2

[Johan Lorenzo [:jlorenzo]]

If I understand things correctly, your current signers are capable of performing either v1 or v2 signing. I.e you code is already setting a mode when doing APK signing.

More or less 🙂 The current signers actually sign APKs with both v1 and v2 signatures[1]. Multiple signature versions can exist on a single APK which enables Android to select the latest it can handle.

Given that, is there a preference on your end to have a separate signer for v3 (which would also be capable of v1 & v2 signing - you'd only get v3 if explicitly requested). If not, I'd prefer to add the v3 enablement to an existing signer, to reduce configuration complexity.

Having separate signers for v3 still signs APKs with v1 and v2 signatures. The benefit we get by creating new signers is: we're going to let the v3 signature ride the trains instead of enabling v3 signatures everywhere at once. For the sake of risk manager, I'd stick to new signers.

Also, could you clarify if you have the ability to test any of this in stage.

From what I see here[3], none of our level-1 workers use the stage instance of Autograph. They all use the production one. I don't know what the complexity is going to be. I remember some very strict firewall rules to access any instance of Autograph.

[1] https://github.com/mozilla-services/autograph/blob/8ddfa3896a2297ce2bbc3e6408853c6dcdb7973e/signer/apk2/apk2.go#L149
[2] https://github.com/mozilla-services/autograph/pull/645#issuecomment-813432999
[3] https://github.com/mozilla-releng/scriptworker-scripts/blob/df0387c2982fc6f0c7075ec236142d06e12838ce/signingscript/docker.d/passwords.yml#L85

Comment 3

[hwine]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)

Given that, is there a preference on your end to have a separate signer for v3 (which would also be capable of v1 & v2 signing - you'd only get v3 if explicitly requested). If not, I'd prefer to add the v3 enablement to an existing signer, to reduce configuration complexity.

Having separate signers for v3 still signs APKs with v1 and v2 signatures. The benefit we get by creating new signers is: we're going to let the v3 signature ride the trains instead of enabling v3 signatures everywhere at once. For the sake of risk manager, I'd stick to new signers.

Followup:

1. Will you want to disable the old signers once the train reach release?
2. Do you want new credentials for the new signers?
3. Please confirm the new signer should use the existing private keys.

Also, could you clarify if you have the ability to test any of this in stage.

From what I see here[3], none of our level-1 workers use the stage instance of Autograph. They all use the production one. I don't know what the complexity is going to be. I remember some very strict firewall rules to access any instance of Autograph.

iirc, the thought wasn't to "run on stage", but rather "allow a manual sign operation to access stage". :bhearsum & :ahal would have the history there, I think

Comment 4

[Johan Lorenzo [:jlorenzo]]

1. Will you want to disable the old signers once the train reach release?

Yes, please 🙂 I can file a follow-up bug, if you wish.

2. Do you want new credentials for the new signers?

I don't have any opinion on this. We rotated credentials a few months ago so it's likely not a big deal if we reuse the same creds - especially given answers for question 1 and 3 🙂

3. Please confirm the new signer should use the existing private keys.

Yes, please. Key rotation on APKs are a thing only starting v3 signatures, so we have to keep the existing keys in order not to break any downstream Android who are stuck on v1 or v2 signatures.

Comment 5

[hwine]

(In reply to Johan Lorenzo [:jlorenzo] from comment #4)

3. Please confirm the new signer should use the existing private keys.

Yes, please. Key rotation on APKs are a thing only starting v3 signatures, so we have to keep the existing keys in order not to break any downstream Android who are stuck on v1 or v2 signatures.

Just noting that this is a potential footgun if we do rotate APK v3 keys, as there is no (current) Autograph restrictions to prevent doing v1 or v2 signing with a v3 key.

Comment 6

[hwine]

Bad news: Autograph configs are hard to reason about
Good news: all the v3 signers were created when v3 support was added back in bug 1703293 (2021-05-03).

Here are the lists of currently authorized signers, using the same credentials as the old signer listed.

Staging:

❯ sops -d type/autograph.app.autographstage.yaml | yq -rc '.authorizations[] | [.id, .signers[] ]' | grep v3\"      ["autograph_edge_v3","dummyapp_android_v3"]
["dummy_legacy_apk_with_rsa","legacy_apk_with_rsa","legacy_apk_with_rsa_v3"]
["releng_signingscript_fenix_dep_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_fenix_nightly_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_fenix_beta_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_fenix_release_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_focus_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_focus_dep_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_focus_rel_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_geckoview_reference_browser_dep_2020","dummyapp_android","dummyapp_android_v3"]
["releng_signingscript_geckoview_reference_browser_rel_2020","dummyapp_android","dummyapp_android_v3"]

Production

❯ sops -d type/autograph.app.autographprod.yaml | yq -rc '.authorizations[] | [.id, .signers[] ]' | grep v3\"
["autograph_edge_v3","send_android_rel_v3","send_android_dep_v3","stumbler_rel_v3","rocket_apk_v3","notes_apk_v3","firefox_connect_apk_v3","screenshot_plus_apk_v3","fx_reality_apk_v3","lockbox_android_v3","fx_lite_preview_dep_v3","firefox_vpn_preview_apk_v3","firefox_vpn_apk_v3","pocket_apk_v3"]
["releng_firefox_tv_dep_2023","firefox_tv_apk_dep","firefox_tv_apk_dep_v3"]
["releng_firefox_tv_rel_2023","firefox_tv_apk_rel","firefox_tv_apk_rel_v3"]
["releng_firefox_vpn_preview_apk_2023","firefox_vpn_preview_apk","firefox_vpn_preview_apk_v3"]
["releng_firefox_vpn_apk_2023","firefox_vpn_apk","firefox_vpn_apk_v3"]
["releng_signingscript_fenix_dep_2023","fenix_dep_apk","fenix_dep_apk_v3"]
["releng_signingscript_fenix_nightly_2023","fenix_nightly_apk","fenix_nightly_apk_v3"]
["releng_signingscript_fenix_beta_2023","fenix_beta_apk","fenix_beta_apk_v3"]
["releng_signingscript_fenix_release_2023","fenix_release_apk","fenix_release_apk_v3"]
["releng_signingscript_fennec_dep_2023","fennec_dep_apk","fennec_dep_apk_v3"]
["releng_signingscript_fennec_rel_2023","fennec_rel_apk","fennec_rel_apk_v3"]
["releng_signingscript_focus_2023","focus_rel_apk","focus_dep_apk","focus_rel_apk_v3","focus_dep_apk_v3"]
["releng_signingscript_focus_dep_2023","focus_dep_apk","focus_dep_apk_v3"]
["releng_signingscript_geckoview_reference_browser_dep_2023","geckoview_reference_browser_dep_apk","geckoview_reference_browser_dep_apk_v3"]
["releng_signingscript_geckoview_reference_browser_rel_2023","geckoview_reference_browser_rel_apk","geckoview_reference_browser_rel_apk_v3"]
["releng_fenix_china_rel_2023","fenix_china_rel_apk","fenix_china_rel_apk_v3"]

I believe that is everything you need, and thus no config changes are needed. Closing as "works for me" -- repopen and clarify if any changes are needed.

Comment 7

[Johan Lorenzo [:jlorenzo]]

Thanks Hal for finding this out! The more we look into it, the more I realize how close we are 😃

I agree, there's no action needed on the autograph side. That said, could you send the _v3 credentials to me? I checked the history of RelEng's SOPS repo and I didn't see any of them between now and bug 1703293.

Comment 8

[Johan Lorenzo [:jlorenzo]]

That said, could you send the _v3 credentials to me? I checked the history of RelEng's SOPS repo and I didn't see any of them between now and bug 1703293.

Sorry, my last comment was confusing. Let me try again 🙂 The current _v3 signers share the same credentials as the current signers, right? If so, that means RelEng has them for the production instance. For the staging one, I don't think we've ever used them. I'm basing this on the fact we likely didn't rotate the staging creds in early 2023 and the fact the staging instance is just used for mar signing at [1].

If my conclusion is correct, could you send me the credentials of the staging instance? Thanks!

[1] https://github.com/mozilla-releng/scriptworker-scripts/blob/6619c2cb93ea243976465a01732bd52661e2b7f3/signingscript/docker.d/passwords.yml

Comment 9

[hwine]

keys for staging signers (comment 6) are in 1Password vault "bz 1842229 c8" shared with bhearsum, ahal, jcristau, & mlarsonweber

Please let me know when credentials have been stored in your system, so I can delete the vault.

Comment 10

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1842229 - APK creds cleanup and add stage autograph (#833)

Comment 11

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1842229 - [signingscript] Remove unused 'AUTOGRAPH_FENNEC' enviro… (#834)

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-services/cloudops-infra] Bug 1842229 - APK creds cleanup and add stage autograph (#5012)

Comment 13

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-services/cloudops-infra] Bug 1842229 - part 6: Add v3 signatures on Autograph stage for Fenix and Focus (#5013)

Comment 14

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1842229 - part 5: Add v3 signatures on Autograph stage for Fenix and Focus (#836)