1827256 - Run Nightly start up benchmarks on geckoview_example

Status: VERIFIED FIXED

(Testing :: mozperftest, enhancement)

Description

[Andrej (:aglavic)]

We currently run Nightly start up benchmarks on fenix and can identify the fenix commit that caused a regression with a local tool. One issue we run into is that sometimes the commit that caused the regression is a GeckoView version upgrade, i.e. the regression came from Gecko but we have no way to bisect to identify the root cause. We’re forced to use more primitive tools (e.g. comparing profiles) to find the root cause.

The 80/20 solution I (mcomella) identified is that we can run the same start up benchmarks on Nightly versions of geckoview_example. This is a partial coverage: we can bisect regressions in always-on features of GeckoView but we won’t see regressions in features of GeckoView that only fenix uses. In order to implement this, we need:

Have CI create Nightly release variants geckoview_example (currently, we only upload Nightly debug variants)

Integrate these Nightly builds with the Nightly regression detection benchmark system

We can either implement this in the backfill system I run every Monday or wait until the perf test team replaces that system and integrate it in the replacement. I propose we wait to implement it in the replacement system unless another GeckoView regression occurs at which point we’ll implement it in the backfill system (i.e. wait to implement it until we need it).

Side note: the “correct” solution to this problem is to have a tool that can bisect mozilla-central commits inside of fenix but that’s quite complex so I do not think we should approach that first.

Comment 1

[Andrej (:aglavic)]

Attachment: Bug 1827256 - Run Nightly start up benchmarks on geckoview_example. r?#perftest

Comment 2

[Andrej (:aglavic)]

Attachment: Bug 1827256 - Run Nightly start up benchmarks on geckoview_example. r?#perftest

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0aecf5129c67
Run Nightly start up benchmarks on geckoview_example. r=perftest-reviewers,sparky

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/0aecf5129c67

Comment 5

[Chris Peterson [:cpeterson]]

Thanks! I verified that the AndroidStartup:geckoview_example test results are now available in Perfherder:

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&series=mozilla-central,4754836,1,15&series=mozilla-central,4754839,1,15&series=mozilla-central,4754838,1,15&series=mozilla-central,4754835,1,15&timerange=2592000