1631558 - [meta] Run additional benchmarks on android

Status: REOPENED

(Testing :: Raptor, task, P3)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We're currently only running speedometer benchmark on mobile. This bug tracks identifying and enabling suitable benchmark tests for running on mobile.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The meta keyword is there, the bug doesn't depend on other bugs and there is no activity for 12 months.
:sparky, maybe it's time to close this bug?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This isn't a duplicate of bug 1642329 as that only migrated the benchmarks running on webext to browsertime. This bug is about considering additional benchmarks to run on Android. For example, we do not currently run ARES-6, JetStream 2, or StyleBench on Android.

Comment 4

[Greg Mierzwinski [:sparky]]

Ok, I've updated the bug title to clarify this.

Comment 5

[Florin Strugariu [:Bebe]]

Andrew Creskey / Michael Comella do you have a list of benchmarks that you would like to see on mobile? Here's what we have set up in Raptor: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks

Comment 6

[Andrew Creskey [:acreskey]]

Sorry, I missed this in the JIRA.

Did we used to run these before?
I thought that we used to use the youtube playback tests as part of the Fenix launch criteria.

That is an extensive list of benchmarks and I'm not that familiar with most of them.
Let me ask for more input from folks on the perf team.

Comment 7

[Andrew Creskey [:acreskey]]

Greg mentioned that the youtube widevine tests are failing.

In general, if we were to look at this from a priority perspective, I would put youtube and speedometer at the top.

But more input is needed because I'm only vaguely aware of what most of those benchmarks do.

If some of them have a tendency to catch improvements or regressions on desktop, I can see how that would make them more valuable.

Comment 8

[Kim Moir [:kmoir] ET]

mcomella I talked to Dave and this request is because the perftest team is reviewing if there’s value running additional benchmarks on mobile. Currently there are many we run on desktop but only a couple on mobile.