2058679 - Add a fenix nightly as release and run sp3 on it

Status: RESOLVED FIXED

(Testing :: Performance, task, P1)

Description

[Kash Shampur [:kshampur] ⌚EST]

follow up from sp3 meeting
not as high priority compared to having desktop but nice to have the numbers example here:

https://perftest-sp3-fenix-nar-vs-nightly.quick.mozilla.cloud/

4-5% faster than nightly, which is kind of expected.

we could run this and only on sp3

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

there is interest in reporting fenix NaR scores so reviving this. couple small todos

• fixup patch since we don't run no-fis variant anymore
• update pretty name in treeherder side
• run on p6 and s24?

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 2058679 - Add a fenix nightly as release build and run SP3 on it r?#perftest

This patch adds an android version of the desktop nightly-as-release
perf config. This gies android the same coverage that we have on macOS,
windows, and linux. This analog ensures we excercise code paths that are
otherwise gated behind nightly only flags.

For now it is restricted to running the Speedometer 3 benchmark.

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

PR for treeherder pretty name https://github.com/mozilla/treeherder/pull/9864

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a236e44ae536
https://hg.mozilla.org/integration/autoland/rev/06d9177f8939
Add a fenix nightly as release build and run SP3 on it r=perftest-reviewers,releng-reviewers,aglavic,ahal,taskgraph-reviewers

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/06d9177f8939

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/bf645b065c8a
https://hg.mozilla.org/mozilla-central/rev/76f2ff0666f3
Revert "Bug 2058679 - Add a fenix nightly as release build and run SP3 on it r=perftest-reviewers,releng-reviewers,aglavic,ahal,taskgraph-reviewers" for causing gecko decision bustages

Comment 7

[Sandor Molnar[:smolnar]]

Backed out for causing gecko decision task crashes

Backout link: https://hg-edge.mozilla.org/mozilla-central/rev/76f2ff0666f3062d8e17abe218205ab1a6564d1f

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

ah i see... dummy secrets are used in Try, m-c uses real secrets? i wonder if fenix NaR can use the same secrets for nightly-sim as regular fenix nightly...

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

hey ahal, wanted to get a sanity check before I reland bug 2058679 (it got backed out for crashing the decision task on mozilla-central due to invalid scope).

so build-apk tasks derive their secret path from the task name, so my new build-apk-fenix-nightlyasrelease-simulation asked for secrets:get:project/mobile/firefox-android/fenix/nightlyasrelease-simulation, which was never provisioned. Only affects level 3 (Try uses dummy secrets on level 1), which is why all my try pushes were green before

in this new commit, my plan is to set include-shippable-secrets: false on that task so it builds with the dummy tokens at every level, same as what Try already does.

I assume this is safe since this APK is a very specific setup: nightly built as release, only used to run Speedometer 3 on lab devices, and never shipped.

I considered pointing it at the existing nightly-simulation secret instead, but it seems that would mix its sentry/adjust/telemetry into the existing nightly-simulation buckets, which I want to avoid.

And provisioning a dedicated secret in fxci-config felt like overkill for this very specific perf-only apk

Any reason a level 3 m-c build like this must have real tokens, or anything else this breaks that I'm not seeing? If dummy secrets are fine I'll fold this in and reland the patch

Comment 10

[Andrew Halberstadt [:ahal]]

We don't need to provision individual secrets, there are buckets for each project such that tasks already have access to any secret in that namespace. For gecko they are:

project/releng/gecko/build/level-1/
project/releng/gecko/build/level-3/

If you don't have access to create secrets there, you can request one of us to add something for you, or we can grant you permission to set a specific secret under those buckets.

But generally speaking, tasks should never use production secrets if they don't need to. If switching the task to use dummy secrets doesn't hinder the goal of the task, then yes, I recommend switching it :)

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4b5920571e7f
https://hg.mozilla.org/integration/autoland/rev/7f52d3e37d8b
Add a fenix nightly as release build and run SP3 on it r=perftest-reviewers,releng-reviewers,aglavic,ahal,taskgraph-reviewers

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7f52d3e37d8b