Bug 1613113 enabled APKv2 signatures by using Google's `apksigner`. This tool takes care of zipaligning APKs because the v2 scheme is encoded in the APK metadata itself. Zipaligning again breaks the metadata. This is what caused today's exception when reference-browser was pushed to Google Play[1]: ``` googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.reference.browser/edits/04263451879600346226/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer APK2_GEC.RSA): JAR signature META-INF/APK2_GEC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?"> ``` Fun fact, Fenix Nightly has been using APKv2 signatures for 2 months (since bug 1489946 comment 19) and it's worked perfectly. Why has it worked on Fenix and not R-B? It turns out `signingscript` does perform a round of zipalign after an APK is signed[2] thanks to the function called `sign_jar`. `sign_jar` is indeed used when the `autograph_apk_.+` signing format is used[3], like in reference-browser[4]. However, Fenix uses a slightly different format: `autograph_apk`, which is not captured by the previous regex and thus, uses the default signer[5]. This default signer doesn't zipalign, that's why Fenix has worked so far! I backtracked how Fenix and R-B became different, and I there is no explicit reason. R-B was introduced first as `autograph_apk_reference_browser`[6] and Fenix came second[7]. Nobody called out the different in the PR. Maybe we did chat about it on another channel, but I have no recollection of it. Anyway, today there are just 2 signing formats that match `autograph_apk_*`: * `autograph_apk_reference_browser`, that we want to fix. * `autograph_apk_fennec_sha1`[8], that is not impacted by bug 1613113. This format hasn't moved over to APKv2 signing, so we want zipalign to keep running. There is another thing that will break: `autograph_focus`. It doesn't match regex but it points to `sign_jar` and it was migrated to APKv2 signing. To sum up: Both `autograph_apk_*` and `autograph_focus` should point to `sign_file` instead of `sign_jar` [1] https://firefox-ci-tc.services.mozilla.com/tasks/ZctwoYJPSsuqPhgPJYSWLQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FZctwoYJPSsuqPhgPJYSWLQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L45 [2] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/sign.py#L204 [3] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L37 [4] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L273 [5] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L53 [6] https://github.com/mozilla-releng/build-puppet/pull/328/files#diff-af5179a901002e9d8713612ac0760248R15 [7] https://github.com/mozilla-releng/build-puppet/pull/360/files#diff-af5179a901002e9d8713612ac0760248R6 [8] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L153 [9] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L36
Bug 1618531 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Bug 1613113 enabled APKv2 signatures by using Google's `apksigner`. This tool takes care of zipaligning APKs because the v2 scheme is encoded in the APK metadata itself. Zipaligning again breaks the metadata. This is what caused today's exception when reference-browser was pushed to Google Play[1]: ``` googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.reference.browser/edits/04263451879600346226/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer APK2_GEC.RSA): JAR signature META-INF/APK2_GEC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?"> ``` Fun fact, Fenix Nightly has been using APKv2 signatures for 2 months (since bug 1489946 comment 19) and it's worked perfectly. Why has it worked on Fenix and not R-B? It turns out `signingscript` does perform a round of zipalign after an APK is signed[2] thanks to the function called `sign_jar`. `sign_jar` is indeed used when the `autograph_apk_.+` signing format is used[3], like in reference-browser[4]. However, Fenix uses a slightly different format: `autograph_apk`, which is not captured by the previous regex and thus, uses the default signer[5]. This default signer doesn't zipalign, that's why Fenix has worked so far! I backtracked how Fenix and R-B became different, and I there is no explicit reason. R-B was introduced first as `autograph_apk_reference_browser`[6] and Fenix came second[7]. Nobody called out the different in the PR. Maybe we did chat about it on another channel, but I have no recollection of it. Anyway, today there are just 2 signing formats that match `autograph_apk_*`: * `autograph_apk_reference_browser`, that we want to fix. * `autograph_apk_fennec_sha1`[8], that is not impacted by bug 1613113. This format hasn't moved over to APKv2 signing, so we want zipalign to keep running. There is another thing that will break: `autograph_focus`. It doesn't match regex but it points to `sign_jar`[9] and it was migrated to APKv2 signing. To sum up: Both `autograph_apk_*` and `autograph_focus` should point to `sign_file` instead of `sign_jar` [1] https://firefox-ci-tc.services.mozilla.com/tasks/ZctwoYJPSsuqPhgPJYSWLQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FZctwoYJPSsuqPhgPJYSWLQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L45 [2] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/sign.py#L204 [3] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L37 [4] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L273 [5] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L53 [6] https://github.com/mozilla-releng/build-puppet/pull/328/files#diff-af5179a901002e9d8713612ac0760248R15 [7] https://github.com/mozilla-releng/build-puppet/pull/360/files#diff-af5179a901002e9d8713612ac0760248R6 [8] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L153 [9] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L36
Bug 1613113 enabled APKv2 signatures by using Google's `apksigner`. This tool takes care of zipaligning APKs because the v2 scheme is encoded in the APK metadata itself. Zipaligning again breaks the metadata. This is what caused today's exception when reference-browser was pushed to Google Play[1]: ``` googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.reference.browser/edits/04263451879600346226/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer APK2_GEC.RSA): JAR signature META-INF/APK2_GEC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?"> ``` Fun fact, Fenix Nightly has been using APKv2 signatures for 2 months (since bug 1489946 comment 19) and it's worked perfectly. Why has it worked on Fenix and not R-B? It turns out `signingscript` does perform a round of zipalign after an APK is signed[2] thanks to the function called `sign_jar`. `sign_jar` is indeed used when the `autograph_apk_.+` signing format is used[3], like in reference-browser[4]. However, Fenix uses a slightly different format: `autograph_apk`, which is not captured by the previous regex and thus, uses the default signer[5]. This default signer doesn't zipalign, that's why Fenix has worked so far! I backtracked how Fenix and R-B became different, and I there is no explicit reason. R-B was introduced first as `autograph_apk_reference_browser`[6] and Fenix came second[7]. Nobody called out the difference in the PR. Maybe we did chat about it on another channel, but I have no recollection of it. Anyway, today there are just 2 signing formats that match `autograph_apk_*`: * `autograph_apk_reference_browser`, that we want to fix. * `autograph_apk_fennec_sha1`[8], that is not impacted by bug 1613113. This format hasn't moved over to APKv2 signing, so we want zipalign to keep running. There is another thing that will break: `autograph_focus`. It doesn't match regex but it points to `sign_jar`[9] and it was migrated to APKv2 signing. To sum up: Both `autograph_apk_*` and `autograph_focus` should point to `sign_file` instead of `sign_jar`. [1] https://firefox-ci-tc.services.mozilla.com/tasks/ZctwoYJPSsuqPhgPJYSWLQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FZctwoYJPSsuqPhgPJYSWLQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L45 [2] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/sign.py#L204 [3] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L37 [4] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L273 [5] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L53 [6] https://github.com/mozilla-releng/build-puppet/pull/328/files#diff-af5179a901002e9d8713612ac0760248R15 [7] https://github.com/mozilla-releng/build-puppet/pull/360/files#diff-af5179a901002e9d8713612ac0760248R6 [8] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L153 [9] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L36
Bug 1613113 enabled APKv2 signatures by using Google's `apksigner`. This tool takes care of zipaligning APKs because the v2 scheme is encoded in the APK metadata. Zipaligning again breaks the metadata. This is what caused today's exception when reference-browser was pushed to Google Play[1]: ``` googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.reference.browser/edits/04263451879600346226/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer APK2_GEC.RSA): JAR signature META-INF/APK2_GEC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?"> ``` Fun fact, Fenix Nightly has been using APKv2 signatures for 2 months (since bug 1489946 comment 19) and it's worked perfectly. Why has it worked on Fenix and not R-B? It turns out `signingscript` does perform a round of zipalign after an APK is signed[2] thanks to the function called `sign_jar`. `sign_jar` is indeed used when the `autograph_apk_.+` signing format is used[3], like in reference-browser[4]. However, Fenix uses a slightly different format: `autograph_apk`, which is not captured by the previous regex and thus, uses the default signer[5]. This default signer doesn't zipalign, that's why Fenix has worked so far! I backtracked how Fenix and R-B became different, and I there is no explicit reason. R-B was introduced first as `autograph_apk_reference_browser`[6] and Fenix came second[7]. Nobody called out the difference in the PR. Maybe we did chat about it on another channel, but I have no recollection of it. Anyway, today there are just 2 signing formats that match `autograph_apk_*`: * `autograph_apk_reference_browser`, that we want to fix. * `autograph_apk_fennec_sha1`[8], that is not impacted by bug 1613113. This format hasn't moved over to APKv2 signing, so we want zipalign to keep running. There is another thing that will break: `autograph_focus`. It doesn't match regex but it points to `sign_jar`[9] and it was migrated to APKv2 signing. To sum up: Both `autograph_apk_*` and `autograph_focus` should point to `sign_file` instead of `sign_jar`. [1] https://firefox-ci-tc.services.mozilla.com/tasks/ZctwoYJPSsuqPhgPJYSWLQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FZctwoYJPSsuqPhgPJYSWLQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log#L45 [2] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/sign.py#L204 [3] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L37 [4] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L273 [5] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L53 [6] https://github.com/mozilla-releng/build-puppet/pull/328/files#diff-af5179a901002e9d8713612ac0760248R15 [7] https://github.com/mozilla-releng/build-puppet/pull/360/files#diff-af5179a901002e9d8713612ac0760248R6 [8] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/docker.d/passwords.yml#L153 [9] https://github.com/mozilla-releng/scriptworker-scripts/blob/f8c77af3d1969daa450df5b2c4882c16ff7178b4/signingscript/src/signingscript/task.py#L36