1826560 - startup-test-nightly-arm failure - signature mismatch between APKs

Status: VERIFIED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Aaron Train [:aaronmt]]

startup-test-nightly-arm https://firefox-ci-tc.services.mozilla.com/tasks/VhDoZxMIRXqhCMJR-6yA2w

Matrix - https://console.firebase.google.com/project/moz-fenix/testlab/histories/bh.80a1a071cfb2770e/matrices/823472642236766067

Permission Denial: starting instrumentation ComponentInfo{org.mozilla.fenix.test/androidx.test.runner.AndroidJUnitRunner} from pid=7803, uid=7803 not allowed because package org.mozilla.fenix.test does not have a signature matching the target org.mozilla.fenix

Comment 1

[Aaron Train [:aaronmt]]

From signing-apk-fenix-android-test-nightly (good) https://firefox-ci-tc.services.mozilla.com/tasks/PjZnnoG_Sj2DDNwT6wegRw

Signer #1 certificate DN: CN=Throwaway Key, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US
Signer #1 certificate SHA-256 digest: 8730ee05de8883b8b60604e74fc2066cc1c17d7a5182b385ed6101186ad4421c
Signer #1 certificate SHA-1 digest: 4822af9ce5768f9b57b2793576ea085dd5b24e8b
Signer #1 certificate MD5 digest: 8efa8fb3f5a0e370c0a56c22b7be6ce4

From signing-apk-fenix-android-test-nightly (bad) https://firefox-ci-tc.services.mozilla.com/tasks/fEx_37JMQHefQlolk_99Aw

Signer #1 certificate DN: CN=Release Engineering, OU=Release Engineering, O=Mozilla Corporation, L=Mountain View, ST=California, C=US
Signer #1 certificate SHA-256 digest: 5004779088e7f988d5bc5cc5f8798febf4f8cd084a1b2a46efd4c8ee4aeaf211
Signer #1 certificate SHA-1 digest: 0cb59840e844e40056ba8f233df1c764a7db4568
Signer #1 certificate MD5 digest: 6ff068cec8762fad9f51758c88289315

Comment 2

[Julien Cristau [:jcristau] (back August 18)]

https://github.com/mozilla-mobile/firefox-android/blob/main/taskcluster/android_taskgraph/transforms/signing_apk.py#L55-L66 looks fairly convoluted and probably broken here.

Johan/Geoff, do you remember why we're signing test apks with a prod key? (PRODUCTION_SIGNING_BUILD_TYPES looks like it includes some test builds)
I don't think it makes sense to look at tasks_for there either.

Comment 3

[Julien Cristau [:jcristau] (back August 18)]

The "bad" apk (using production signing) came out of a cron graph, the "good" one (dep-signed) came from a github-push graph.

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

(In reply to Julien Cristau [:jcristau] from comment #2)

https://github.com/mozilla-mobile/firefox-android/blob/main/taskcluster/android_taskgraph/transforms/signing_apk.py#L55-L66 looks fairly convoluted and probably broken here.

I agree. We used to handle this logic in the kind.yml but this was changed in fenix#18191. Based on this context, I think it's okay to put this logic back in kind.yml.

Johan/Geoff, do you remember why we're signing test apks with a prod key? (PRODUCTION_SIGNING_BUILD_TYPES looks like it includes some test builds)

Yes! This was introduced because of this issue[1]. android-test variants are actually a helper APK we submit to end-to-end test the real APK. It has to be signed with the same key as the APK under test. If I remember correctly, we take the real nightly and we just make sure we're able to start it. We did have crasher that only occurred on nightly in the early days of Fenix. We don't want this to happen again.

[1] https://github.com/mozilla-mobile/fenix/pull/6569#discussion_r350054573

https://github.com/mozilla-mobile/fenix/pull/6569#discussion_r350054573

Comment 6

[Intermittent Failures Robot]

1 failures in 3386 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• firefox-android: 1

Platform and build breakdown:

• nightly-start-test: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1826560&startday=2023-04-03&endday=2023-04-09&tree=all

Comment 7

[Intermittent Failures Robot]

6 failures in 3206 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• firefox-android: 6

Platform and build breakdown:

• nightly-start-test: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1826560&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 8

[Aaron Train [:aaronmt]]

Hey folks, how can this be prioritized?

Comment 9

[Geoff Brown [:gbrown]]

If I'm following this correctly, startup-test-nightly-arm fails when run by the nightly cron because the two apks are not signed the same way.

Following the example of comment 0,
https://firefox-ci-tc.services.mozilla.com/tasks/VhDoZxMIRXqhCMJR-6yA2w/runs/0/logs/public/logs/live.log

[task 2023-04-05T04:29:21.941Z] executing ['bash', '-cx', "cd fenix && ../taskcluster/scripts/get-secret.py -s project/mobile/firefox-android/fenix/firebase -k firebaseToken -f .firebase_token.json --json && wget 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DP4Ac1WHTkmaQj-YrBehWw/artifacts/public/build/fenix/arm64-v8a/target.apk' -O app.apk && wget 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fEx_37JMQHefQlolk_99Aw/artifacts/public/build/fenix/noarch/target.apk' -O android-test.apk && automation/taskcluster/androidTest/ui-test.sh arm-start-test app.apk android-test.apk 1"]

https://firefox-ci-tc.services.mozilla.com/tasks/DP4Ac1WHTkmaQj-YrBehWw is a signing-apk-fenix-nightly-firebase task that uses dep-signing.
https://firefox-ci-tc.services.mozilla.com/tasks/fEx_37JMQHefQlolk_99Aw is a signing-apk-fenix-android-test-nightly task that uses production-signing.

That seems consistent with PRODUCTION_SIGNING_BUILD_TYPES: fenix-android-test-nightly is there, but not fenix-nightly-firebase.

I would prefer to remove fenix-android-test-nightly from PRODUCTION_SIGNING_BUILD_TYPES, but I feel like it's needed (comment 4, even though I think that's ambiguous); therefore can we simply add fenix-nightly-firebase to PRODUCTION_SIGNING_BUILD_TYPES?

Comment 10

[Julien Cristau [:jcristau] (back August 18)]

To sum up, there's a bit of a split head thing here where:

• we run build-apk-fenix-nightly-firebase both on push and cron, and sign it with a dep key both times
• we run build-apk-fenix-android-test-nightly from cron, and sign it with a prod key
• we run build-apk-fenix-android-test-nightly on push, and sign it with a dep key
• we run startup-test-nightly-arm from cron, which expects signing-apk-fenix-nightly-firebase and signing-apk-fenix-android-test-nightly to use the same key.

I guess one way to fix it is indeed to add fenix-nightly-firebase to PRODUCTION_SIGNING_BUILD_TYPES, whereby the cron graph would use prod signing for that apk. However IMO doing different things for the same task on push vs cron is an anti-pattern, so I'd prefer to get rid of that rather than add more of it.

[Ideally we'd also not schedule those tasks twice on the same push, e.g. like the gecko nightly cron decision task uses the include_push_tasks option.]

Johan: re comment 4, I'm still a bit confused. We don't seem to run any of these tests against the "real" (production) APK, we run them against the fenix-nightly-firebase build. Does anything speak against signing both the android-test apk and the *-firebase apks with dep keys?

Comment 11

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1826560 - consistently sign android-test APKs with the dep key (#1699)

Comment 12

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

(In reply to Johan Lorenzo [:jlorenzo] from comment #4)

If I remember correctly, we take the real nightly and we just make sure we're able to start it.

Sorry for the confusion, I've just realized bug 1819153 actually changed this behavior. We're not testing the real nightly anymore. We switched to the firebase one. The timing actually indicate this is a direct regression.

I just checked the current code base and we're not using any "real" (production) APK anymore. We're just using the firebase ones.

(In reply to Julien Cristau [:jcristau] from comment #10)

Does anything speak against signing both the android-test apk and the *-firebase apks with dep keys?

You're right, there's no need to sign the android-test APKs with the prod keys anymore. Let's just sign all firebase and android-test APKs with the dep keys.

Comment 13

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1819153

Comment 14

[BMO Github Automation]

Authored by https://github.com/jcristau
https://github.com/mozilla-mobile/firefox-android/commit/5faf3b041bbdf1e432bd97ef944e36c6dc5fd6cc
[main] Bug 1826560 - consistently sign android-test APKs with the dep key

Comment 15

[Julien Cristau [:jcristau] (back August 18)]

I think this only(?) affects main, so no need to uplift.

Comment 16

[Aaron Train [:aaronmt]]

Thanks all, I see green on startup-arm again.
https://firefox-ci-tc.services.mozilla.com/tasks/TmF4jC-qSTiSI-pSyVA2dw

Comment 17

[Intermittent Failures Robot]

7 failures in 4004 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• firefox-android: 7

Platform and build breakdown:

• nightly-start-test: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1826560&startday=2023-04-17&endday=2023-04-23&tree=all