1646368 - Add mobile device performance tuning to perf test

Status: RESOLVED FIXED

(Testing :: mozperftest, enhancement)

Description

[Andrew Creskey [:acreskey]]

Other android tests that run in bitbar make use of performance tuning logic that limits CPU variability.

This bug is to add this tuning to the android perf tests.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1646368 - Add performance tuning feature to android layer. r?#mozperftest-reviewers

This patch ports the performance tuning functionality from Raptor to mozperftest and adds a small test for it.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db91db5db352
Add performance tuning feature to android layer. r=mozperftest-reviewers,tarek

Comment 3

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/db91db5db352

Comment 4

[Andrew Creskey [:acreskey]]

This is interesting, it appears that the introduction of performance tuning is really slowing down G5 applink times.

Baseline (with perf tuning): App-link opt: 3223.7
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6867bbb536436aa885daf2fde18074491f8ff321&selectedTaskRun=FgTHYvyxQxq_bx5s68RpDA.0

With perf tuning disabled: App-link opt: 2781.8
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=Jm_ntYmgRhKV6dFOJtWx1A.0&tier=1%2C2%2C3&revision=fd42d87cd33e17402541e8cf68b5ebe43da1fecb

I'll double check that this effect is reproducible.

Comment 5

[Andrew Creskey [:acreskey]]

Reproduced.
I guess it surprises me that the the final result is so much slower.
But it should be more predictable (I haven't looked at the noise yet).

Baseline (with perf tuning): App-link opt: 3211.4
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=76bd92f6179797a29258f84ee7d4d627835ee2bc&selectedTaskRun=SOxRsfSYSe-i5vwBfg_IMQ.0

With perf tuning disabled: App-link opt: 2785.9
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=8eb565c4e89caf52373e82558f2a0fde94370d29&selectedTaskRun=S27WMLvTReWHlRIrBFwraA.0

Comment 6

[Greg Mierzwinski [:sparky]]

:acreskey, from the graphs I made in phab, it looks like the noise is really bad atm. :egao, I recall that you implemented the tuning feature for android tests, would you have any thoughts on what we can do here for G5? It seems like the tuning might be making the data worse.

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

Yes, I did implement the tuning features in Raptor suites, though I admit I am not surprised that Motorola G5 units are not behaving as well as they should be.

I tuned the Pixel 2 with a local hardware I had on hand, so I was able to experiment quickly and where integer values were required (eg. frequency) I could get them quickly.

I did not have a Motorola G5 on hand, so I had to try them on tryserver and this was a pain.

Let's evaluate if the perf tuning is still meeting our requirements. Is there a way to see the performance trendeline for the applink on both fenix and fennec going back a year? If not, let's run 20+ instances of a handful of suites and find out the variant with and without the tuning.

It is possible that while device tuning worked well last year this time, doing so caused extra wear and tear on the device as one possible explanation of why we're seeing worse results. The other possible explanation is that fenix behaves much lighter and better than fennec does, so requires less fiddling with the hardware.

Comment 8

[Greg Mierzwinski [:sparky]]

Ok, I've started another bug for this and we can continue the conversation there.