1513564 - Fennec beta automation busted, can't publish the APK because of broken signing

Status: VERIFIED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Pushing the APK failed in 65.0b4. This is the first Fennec we have rolled-out with Autograph scopes. 

Scopes look good in signing jobs (e.g. this[0]).It's failing as Google API rejects the signature[1]:

    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.firefox_beta/edits/09543169524220030044/apks?alt=json&uploadType=media returned "APK signature is invalid or does not exist. Error from apksigner: ERROR (Jar signer SIGNATURE.RSA): Failed to verify JAR signature META-INF/SIGNATURE.RSA against META-INF/SIGNATURE.SF: java.security.SignatureException: APKs with Signed Attributes broken on platforms with API Level < 19">


[0]: https://tools.taskcluster.net/groups/BI2JX0aJTN6UDG8vC1JE3w/tasks/BQzlB2dpTQW6jM07Mia57g/details
[1]: https://taskcluster-artifacts.net/QMrpgSC2SUO4g8PUrFpD7A/0/public/logs/live_backing.log

Comment 1

[Jordan Lund (:jlund)]

removing 65.0b4 from title as that is no longer blocked. We manually signed and published the apk.

For this bug we either need to:

1) fix autograph
2) revert to old signing-server infra

Comment 2

[Jordan Lund (:jlund)]

Greg, assigning to you for now as we try (1) fixing autograph first. Feel free to 302 as appropriate.

Comment 3

[Greg G]

(In reply to Jordan Lund (:jlund) from comment #2)
> Greg, assigning to you for now as we try (1) fixing autograph first. Feel
> free to 302 as appropriate.

For autograph :ulfr is planning to submit a patch tomorrow [0] with his local changes. There's a change freeze in place, but we can deploy that fix the week of Jan. 7th (prod should be out Wednesday the 9th).

:jlund Do we need to sign another Fennec beta or release build before Jan. 9th? I could see about getting the OK for a bugfix deploy.


[0] :ulfr from #autograph IRC scrollback today:

> g-k: I'll write a patch, probably tomorrow morning, to make the apk signer configurable and use different OIDs and signing functions from the pkcs7 library when set to legacy mode.
> I don't think this needs to be an option the client passes
> ...
> so, the tldr for posterity is: old android requires old pkcs7 oids and no signed attributes.

Comment 4

[Jordan Lund (:jlund)]

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #3)

> :jlund Do we need to sign another Fennec beta or release build before Jan.
> 9th? I could see about getting the OK for a bugfix deploy.
> 

We have 4 fennec releases scheduled *before* Jan 9th[0]

So we will need to either revert signing infra or get an approval for autograph infra deploy.

Greg: Given that the freeze just happened and we are still a bit out from holidays, I'm hoping we can make an exception to the freeze and unblock busted release automation.

Thanks for attacking this bug so quickly.


[0] https://calendar.google.com/calendar/embed?src=bW96aWxsYS5jb21fZGJxODRhbnI5aTh0Y25taGFiYXRzdHY1Y29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

Comment 5

[Greg G]

(In reply to Jordan Lund (:jlund) from comment #4)
> (In reply to Greg Guthe [:g-k] [:gguthe] from comment #3)
> 
> Greg: Given that the freeze just happened and we are still a bit out from holidays, I'm hoping we can make an exception to the freeze and unblock busted release automation.

Yeah definitely, I emailed :habib and :miles about making an exception to deploy a fix. We're looking to deploy prod next Wednesday, but can try for earlier in the week if beta 5 needs to be signed before then.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

65.0b5 go-to-build is scheduled to happen on Monday with the push to the Play Store on Tuesday. My understanding is that signing happens when we do the push to the Play Store, so doing it on Tuesday would be better from my perspective. I can certainly hold off on doing the actual push until I get your go-ahead.

Comment 7

[Jordan Lund (:jlund)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #6)
> 65.0b5 go-to-build is scheduled to happen on Monday with the push to the
> Play Store on Tuesday. My understanding is that signing happens when we do
> the push to the Play Store, so doing it on Tuesday would be better from my
> perspective. I can certainly hold off on doing the actual push until I get
> your go-ahead.

sorry, by on-push I mean push to hg (per-checkin builds): https://tools.taskcluster.net/groups/BI2JX0aJTN6UDG8vC1JE3w/tasks/LFlbjwMvSMqfw09r5Uk48A/details

Which means we either need to:

1) deploy on Mon as part of the initial build + test rev
2) revert signing infra by end of week.
3) manually publish next week's beta

Comment 8

[Greg G]

I got approval from :habib (on PTO this week) to do an emergency bugfix deploy during the change freeze and I think we can deploy prod on Monday.

I still have to get :ulfr's patch in and confirm ops coverage, but I think we can assume autograph will be ready in time.

Comment 9

[Johan Lorenzo [:jlorenzo]]

(In reply to Jordan Lund (:jlund) from comment #7)
> Which means we either need to:
> 
> [...]
> 2) revert signing infra by end of week.

In case we go with this solution, it's a matter of reverting bug 1492245 comment 24 on mozilla-beta, then go-to-build. There is no other action needed.

Comment 10

[Greg G]

So we were able to get a fix out faster than I expected (especially considering :habib is on PTO and :miles is at kubecon).

Got approval from :habib, landed :ulfr's PR to fix the issue [0], and :miles fixed a deploy bug [1] and deployed stage and prod today.

Resolving this, but we should reopen if Fennec signing is still busted.

[0] https://github.com/mozilla-services/autograph/pull/194
[1] https://github.com/mozilla-services/cloudops-deployment/pull/2828

Comment 11

[Greg G]

testing against the prod config locally:

wget https://archive.mozilla.org/pub/mobile/releases/64.0b9/android-api-16/en-US/fennec-64.0b9.en-US.android-arm.apk
zip -d fennec-64.0b9.en-US.android-arm.apk 'META-INF/*'
mv fennec-64.0b9.en-US.android-arm.apk fennec-64.0b9.en-US.android-arm.unsigned.apk
go run client.go -f fennec-64.0b9.en-US.android-arm.unsigned.apk -o fennec-64.0b9.en-US.android-arm.signed-sha1.apk -u signingscript_fennec_rel -p $TOKEN -pk7digest SHA1
apksigner verify --print-certs --verbose fennec-64.0b9.en-US.android-arm.signed-sha1.apk
Verifies
Verified using v1 scheme (JAR signing): true
Verified using v2 scheme (APK Signature Scheme v2): false
Number of signers: 1
Signer #1 certificate DN: CN=Release Engineering, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US
Signer #1 certificate SHA-256 digest: a78b62a5165b4494b2fead9e76a280d22d937fee6251aece599446b2ea319b04
Signer #1 certificate SHA-1 digest: 920f4876a6a57b4a6a2f4ccaf65f7d29ce26ff2c
Signer #1 certificate MD5 digest: b1e1bcee2733025ece9456e419a814a3
Signer #1 key algorithm: RSA
Signer #1 key size (bits): 2048
Signer #1 public key SHA-256 digest: aac36655156876c894b09d512e74a0c7bb3301a1c871f200820d099ef1520ae1
Signer #1 public key SHA-1 digest: 3d4fff78408b159058428bff8d07589a4c18aa8b
Signer #1 public key MD5 digest: fe669db38a1eae305b8573293e5c35d8

https://send.firefox.com/download/fa414da726/#xC44X77MOe4XnFLCcS4d4Q password: ced42ccb113157b0 expires in 20 downloads or 24h

:jlorenzo can you test uploading that APK to a dev channel? or if you have a way to test signing against prod without publishing that'd be even better

Comment 12

[Johan Lorenzo [:jlorenzo]]

Thanks a lot Greg and Julien for diagnosing the problem, and putting the patch live super quickly!

Bad news first: I cannot successfully upload the APK in comment 11. The reason is: it's an old one. Google Play prevents replay attacks by enforcing an ever increasing version code. I could cheat by uploading the APK onto a different product, but Google Play gets the product and the version code from the APK itself. 

Good news:
* the original bug was actually caught by apksigner. I was able to reproduce the error locally with it. I retried to repro with the APK in comment 11, and it successfully passed!
* due to a misconfiguration in the taskgraph, Mihai and I discovered we actually sign APKs on every push on mozilla-beta. I checked [1] and the signaturepassed jarsigner and apksigner.

To sum up: I think we're ready for the next beta. 

[1] https://tools.taskcluster.net/groups/V3NByoV4ToS2csbS9uM5zQ/tasks/cCHUBI-kTsOkMSI6EH22Lg/runs/0/artifacts

Comment 13

[Johan Lorenzo [:jlorenzo]]

New signature passed on beta! https://tools.taskcluster.net/groups/ftBEBkD6TmimkteToooUZQ/tasks/P4R0hJSfR8GbXKw2NDE4rw/runs/0/logs/public%2Flogs%2Flive_backing.log#L481