2033466 - prototype a navigation benchmark test

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P3)

Description

[Kash Shampur [:kshampur] ⌚EST]

I think this should be done in the mozperftest harness

This bug will just track that attempt/proof-of-concept and provide some Try runs etc before we expand on the idea.

The idea is similar to the existing navigation tests in raptor however we do this instead by actually "navigating" with things like clicking links rather than explicitly changing URL's

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

correction to comment 0... for the prototype i'll probably keep it in raptor, and locally. Based on some work in Bug 1973522, and discussion with :acreskey

But this bug i'll make a point to emphasize it's just a prototype. One we've determined it has value i'll file another bug where we get it in CI and fully in production

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

(clarified name)

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

we decided in a meeting to land a proxy mode version

here is an example Try run

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=673acbba80a8d1640f7d507ba53a9cbc1eba5bef

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

examining the results from those retriggers

  Platform                              N    Mean    Stdev     CV%     Min      Max
  ----------------------------------------------------------------------------------
  linux2404-64-shippable               22   81.99   1.065    1.30%   80.10    83.94
  macosx1500-aarch64-shippable         18  169.63   2.631    1.55%  164.63   174.87
  macosx1470-64-shippable              18   79.54   2.716    3.41%   74.69    84.31
  windows11-64-24h2-shippable          17   59.34   9.936   16.74%   30.25    67.92

this is what i expected (minus linux). Windows is known to be super noisy at the moment, and we expected MacOS arm64 to be really good.

Surprised that linux is so stable given the recent bimodality!

we probably shouldn't run this on macOS 14.7 but nice to see it was stable too

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

right now since windows shippable is having issues, i think i will consider running this on the hw-ref variant in parallel

additionally i've more or less decided to not run this on 14.7 macOS

WIP is almost ready. one thing is i rebased it on top of Bug 2014503
I'm going to tease bits of that out of it before putting this up for review

so for the time being we will run it on autoland without alerting (tier 3) on:

• windows 11
• Windows 11 (ref hardware)
• macOS 1500 (m4)
• Linux 24.04

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 2033466 - Add a prototype navigation benchmark test. r?#perftest

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/7ac86cb4b484
https://hg.mozilla.org/integration/autoland/rev/3f153d68af59
Add a prototype navigation benchmark test. r=perftest-reviewers,mozperftest-reviewers,sparky

Comment 8

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/3f153d68af59

Comment 9

[Andrew Creskey [:acreskey]]

I've started looking at the Nav Bench in CI

Perf-compare against the same revision, i.e. A/A test

The reproducibility of results looks quite good; overall scores with quite a bit less than 1% difference between pushes (excluding the Windows 11 machines which are known to be a source of noise due to power supplies).

The individual subtests also show the same < 1% variance, with 10 iterations.

Comment 10

[Andrew Creskey [:acreskey]]

Attachment: Nav-bench-aa.png

Comparing the same revision against itself, 10 iterations

Comment 11

[Andrew Creskey [:acreskey]]

Running a few more tests on the nav-bench to get a better understanding of noise and ability to detect regressions.

1.
A new A/A test, i.e. the same patch tested against itself to gauge variance.
PerfCompare

Overall delta < 1%

Benchmark score (higher is better), B:10 N:10. All deltas are Low confidence (within noise).

Platform                        Base      New     Change   Confidence
------------------------------  -------  -------  -------  ----------
Linux (opt fission webrender)    98.37    98.58   +0.21%   Low
macOS 14.70                      95.21    95.74   +0.56%   Low
macOS 15                        190.83   191.25   +0.22%   Low
Windows 11 ref                   87.25    87.22   -0.03%   Low
Windows 11                      115.27   114.19   -0.93%   Low

2.
Introducing a 1ms sleep just before network requests start the connect process, AsyncOpen

PerfCompare

Regression picked up on all platforms but Linux (which curiously was not affected). (desktop)

nav-bench-overall — try push 6c98cca49c6c
Benchmark score (higher is better), B:10 N:10.

Platform                        Base      New     Change   Verdict
------------------------------  -------  -------  --------  ----------
Linux (opt fission webrender)    98.37    97.77   -0.50%   No change
macOS 14.70                      95.21    84.95  -10.86%   Regression
macOS 15                        190.83   108.12  -43.24%   Regression
Windows 11 ref                   87.25    53.57  -38.83%   Regression
Windows 11                      115.27    59.91  -48.03%*  Regression

3.
Introducing at 2ms sleep in AsyncOpen.

Regression picked up on all platforms (desktop).

PerfCompare

Platform                        Base      New     Change   Verdict
------------------------------  -------  -------  --------  ----------
Linux (opt fission webrender)    98.37    94.58   -3.85%   Regression
macOS 14.70                      95.21    71.19  -25.22%   Regression
macOS 15                        190.83    74.55  -60.93%   Regression
Windows 11 ref                   87.25    53.37  -38.83%   Regression
Windows 11                      115.27    60.12  -47.85%   Regression

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

thanks for sharing, i've linked this to the meta bug 2057233 where it might be more visible :D