1618531 - reference-browser cannot be uploaded to the Google Play store anymore: APK Signature Scheme v2 was stripped

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Bug 1613113 enabled APKv2 signatures by using Google's apksigner. This tool takes care of zipaligning APKs because the v2 scheme is encoded in the APK metadata. Zipaligning again breaks the metadata. This is what caused today's exception when reference-browser was pushed to Google Play[1]:

googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.reference.browser/edits/04263451879600346226/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer APK2_GEC.RSA): JAR signature META-INF/APK2_GEC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?">

Fun fact, Fenix Nightly has been using APKv2 signatures for 2 months (since bug 1489946 comment 19) and it's worked perfectly. Why has it worked on Fenix and not R-B?

It turns out signingscript does perform a round of zipalign after an APK is signed[2] thanks to the function called sign_jar. sign_jar is indeed used when the autograph_apk_.+ signing format is used[3], like in reference-browser[4]. However, Fenix uses a slightly different format: autograph_apk, which is not captured by the previous regex and thus, uses the default signer[5]. This default signer doesn't zipalign, that's why Fenix has worked so far!

I backtracked how Fenix and R-B became different, and I there is no explicit reason. R-B was introduced first as autograph_apk_reference_browser[6] and Fenix came second[7]. Nobody called out the difference in the PR. Maybe we did chat about it on another channel, but I have no recollection of it.

Anyway, today there are just 2 signing formats that match autograph_apk_*:

• autograph_apk_reference_browser, that we want to fix.
• autograph_apk_fennec_sha1[8], that is not impacted by bug 1613113. This format hasn't moved over to APKv2 signing, so we want zipalign to keep running.

There is another thing that will break: autograph_focus. It doesn't match regex but it points to sign_jar[9] and it was migrated to APKv2 signing.

To sum up: Both autograph_apk_* and autograph_focus should point to sign_file instead of sign_jar.

[1] https://firefox-ci-tc.services.mozilla.com/tasks/ZctwoYJPSsuqPhgPJYSWLQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FZctwoYJPSsuqPhgPJYSWLQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L45
[2] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/sign.py#L204
[3] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L37
[4] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L273
[5] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L53
[6] https://github.com/mozilla-releng/build-puppet/pull/328/files#diff-af5179a901002e9d8713612ac0760248R15
[7] https://github.com/mozilla-releng/build-puppet/pull/360/files#diff-af5179a901002e9d8713612ac0760248R6
[8] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L153
[9] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L36

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] Let Autograph handle zipaligning in all APK products but Fennec

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] Rename APK signing format

Comment 3

[Johan Lorenzo [:jlorenzo]]

Fixed! The latest nightly has been published to the Google Play Store: https://firefox-ci-tc.services.mozilla.com/tasks/ODnVAXO3Rze9GbwqqN_K_Q/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FODnVAXO3Rze9GbwqqN_K_Q%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L42

Comment 4

[Intermittent Failures Robot]

2 failures in 5233 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• reference-browser: 2

Platform breakdown:

• android: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1618531&startday=2020-02-24&endday=2020-03-01&tree=all