Closed Bug 1646368 Opened 3 months ago Closed 3 months ago

Add mobile device performance tuning to perf test

Categories

(Testing :: mozperftest, enhancement)

enhancement

Tracking

(firefox79 fixed)

RESOLVED FIXED
mozilla79
Tracking Status
firefox79 --- fixed

People

(Reporter: acreskey, Assigned: tarek)

References

Details

Attachments

(1 file)

Other android tests that run in bitbar make use of performance tuning logic that limits CPU variability.

This bug is to add this tuning to the android perf tests.

This patch ports the performance tuning functionality from Raptor to mozperftest and adds a small test for it.

Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/db91db5db352
Add performance tuning feature to android layer. r=mozperftest-reviewers,tarek
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla79

This is interesting, it appears that the introduction of performance tuning is really slowing down G5 applink times.

Baseline (with perf tuning): App-link opt: 3223.7
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6867bbb536436aa885daf2fde18074491f8ff321&selectedTaskRun=FgTHYvyxQxq_bx5s68RpDA.0

With perf tuning disabled: App-link opt: 2781.8
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=Jm_ntYmgRhKV6dFOJtWx1A.0&tier=1%2C2%2C3&revision=fd42d87cd33e17402541e8cf68b5ebe43da1fecb

I'll double check that this effect is reproducible.

Flags: needinfo?(gmierz2)

Reproduced.
I guess it surprises me that the the final result is so much slower.
But it should be more predictable (I haven't looked at the noise yet).

Baseline (with perf tuning): App-link opt: 3211.4
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=76bd92f6179797a29258f84ee7d4d627835ee2bc&selectedTaskRun=SOxRsfSYSe-i5vwBfg_IMQ.0

With perf tuning disabled: App-link opt: 2785.9
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=8eb565c4e89caf52373e82558f2a0fde94370d29&selectedTaskRun=S27WMLvTReWHlRIrBFwraA.0

:acreskey, from the graphs I made in phab, it looks like the noise is really bad atm. :egao, I recall that you implemented the tuning feature for android tests, would you have any thoughts on what we can do here for G5? It seems like the tuning might be making the data worse.

Flags: needinfo?(gmierz2) → needinfo?(egao)

Yes, I did implement the tuning features in Raptor suites, though I admit I am not surprised that Motorola G5 units are not behaving as well as they should be.

I tuned the Pixel 2 with a local hardware I had on hand, so I was able to experiment quickly and where integer values were required (eg. frequency) I could get them quickly.

I did not have a Motorola G5 on hand, so I had to try them on tryserver and this was a pain.

Let's evaluate if the perf tuning is still meeting our requirements. Is there a way to see the performance trendeline for the applink on both fenix and fennec going back a year? If not, let's run 20+ instances of a handful of suites and find out the variant with and without the tuning.

It is possible that while device tuning worked well last year this time, doing so caused extra wear and tear on the device as one possible explanation of why we're seeing worse results. The other possible explanation is that fenix behaves much lighter and better than fennec does, so requires less fiddling with the hardware.

Flags: needinfo?(egao)
See Also: → 1649511

Ok, I've started another bug for this and we can continue the conversation there.

You need to log in before you can comment on or make changes to this bug.