Create an android version of the prototype navigation benchmark
Categories
(Testing :: Performance, task, P2)
Tracking
(firefox161 fixed)
| Tracking | Status | |
|---|---|---|
| firefox161 | --- | fixed |
People
(Reporter: kshampur, Assigned: kshampur)
References
(Blocks 2 open bugs)
Details
(Whiteboard: [fxp][vision][est-4w])
Attachments
(1 external file)
The desktop version (bug 2033466) has been looking quite promising
We should first settle on a list of 5-7 sites and create an equivalent. Realistically speaking though we'll probably have to come up with 10+ sites and choose the best 5-7 performing ones (since not everything records and replays cleanly...)
|
||
Updated•5 months ago
|
|
Assignee | |
Comment 1•3 months ago
•
|
||
(accidentally running on bit bar not LT)
At 5 browser cycles, takes about ~40 minutes to run.
Score (higher = better), CoV% in parentheses
| Metric | Fenix · fission (n=16) | Fenix · nofis (n=17) | GeckoView · fission (n=18) | GeckoView · nofis (n=14) |
|---|---|---|---|---|
| OVERALL | 91.43 (1.7%) | 91.51 (1.7%) | 87.44 (1.2%) | 91.41 (1.4%) |
| amazon-load | 60.93 (2.1%) | 61.50 (2.8%) | 60.97 (2.7%) | 61.48 (2.9%) |
| bbc-load | 42.60 (3.6%) | 42.68 (4.3%) | 40.96 (3.5%) | 42.65 (2.2%) |
| duckduckgo-load | 67.11 (1.8%) | 68.09 (1.8%) | 65.09 (2.6%) | 67.84 (2.8%) |
| reddit-load | 85.94 (3.7%) | 86.13 (5.5%) | 83.54 (4.5%) | 93.56 (4.5%) |
| wikipedia-load | 121.43 (5.7%) | 120.20 (4.8%) | 105.03 (4.3%) | 123.83 (3.7%) |
| amazon-subnav | 142.87 (3.6%) | 141.11 (2.6%) | 133.87 (2.6%) | 133.15 (3.9%) |
| duckduckgo-subnav | 143.55 (2.2%) | 142.93 (1.8%) | 128.67 (2.8%) | 128.05 (3.4%) |
| wikipedia-subnav | 131.43 (4.0%) | 132.27 (2.8%) | 139.57 (2.8%) | 139.13 (3.4%) |
the overall score seems to be pretty low variance, with some noisy subtests...
i'll wait for the LT run to finish to see those results as well
|
|
Assignee | |
Comment 2•3 months ago
|
||
|
|
Assignee | |
Comment 3•3 months ago
•
|
||
Quick update:
1. Baseline variance link
Fenix, fission only, 15 retriggers. Load-only suite at this point (5 sites, no subnav).
| Metric | mean | stdev | CoV% | min | max |
|---|---|---|---|---|---|
| OVERALL | 63.53 | 0.86 | 1.36 | 61.91 | 65.18 |
| amazon | 57.85 | 1.54 | 2.67 | 55.43 | 61.10 |
| bbc | 39.47 | 1.34 | 3.39 | 37.35 | 42.87 |
| duckduckgo | 57.70 | 1.71 | 2.96 | 54.39 | 61.14 |
| 73.14 | 2.50 | 3.41 | 69.39 | 77.36 | |
| wikipedia | 107.69 | 5.33 | 4.95 | 100.99 | 119.42 |
2. Fission on/off link
Subnav subtests were added at this revision (amazon, duckduckgo, wikipedia have subnav; bbc and reddit are load-only). Score (CoV%):
| Metric | Fenix·fission (n=16) | Fenix·nofis (n=17) | GV·fission (n=18) | GV·nofis (n=14) |
|---|---|---|---|---|
| OVERALL | 91.43 (1.7%) | 91.51 (1.7%) | 87.44 (1.2%) | 91.41 (1.4%) |
| amazon-load | 60.93 (2.1%) | 61.50 (2.8%) | 60.97 (2.7%) | 61.48 (2.9%) |
| bbc-load | 42.60 (3.6%) | 42.68 (4.3%) | 40.96 (3.5%) | 42.65 (2.2%) |
| duckduckgo-load | 67.11 (1.8%) | 68.09 (1.8%) | 65.09 (2.6%) | 67.84 (2.8%) |
| reddit-load | 85.94 (3.7%) | 86.13 (5.5%) | 83.54 (4.5%) | 93.56 (4.5%) |
| wikipedia-load | 121.43 (5.7%) | 120.20 (4.8%) | 105.03 (4.3%) | 123.83 (3.7%) |
| amazon-subnav | 142.87 (3.6%) | 141.11 (2.6%) | 133.87 (2.6%) | 133.15 (3.9%) |
| duckduckgo-subnav | 143.55 (2.2%) | 142.93 (1.8%) | 128.67 (2.8%) | 128.05 (3.4%) |
| wikipedia-subnav | 131.43 (4.0%) | 132.27 (2.8%) | 139.57 (2.8%) | 139.13 (3.4%) |
3. Browser cycles
5 BC vs 9 BC, reddit dropped in both, 15/15 Fenix, same config otherwise:
5 BC (cf6363f4) |
9 BC (fb19d21a) |
|
|---|---|---|
| Fenix·fission overall | 79.25 | 79.26 |
| Fenix·fission CoV | 2.50% | 2.47% |
| Fenix·nofis CoV | 1.69% | 2.45% |
| Median runtime | 32.2 min | 54.6 min |
Within-push check — same 15 nine-cycle Fenix jobs, CoV of geomean using first k cycles (removes all push-to-push variation):
| subtest | CoV@5 | CoV@7 | CoV@9 | within-job CoV |
|---|---|---|---|---|
| wikipedia-load | 5.28 | 4.85 | 4.01 | 8.88 |
| amazon-subnav | 4.28 | 3.73 | 3.57 | 5.63 |
| bbc-load | 4.19 | 3.87 | 3.56 | 6.40 |
| wikipedia-subnav | 2.92 | 2.25 | 2.34 | 5.23 |
| amazon-load | 4.82 | 5.07 | 4.67 | 6.42 |
| duckduckgo-load | 3.21 | 3.01 | 3.09 | 3.85 |
| duckduckgo-subnav | 4.38 | 4.59 | 4.63 | 5.36 |
| OVERALL | 2.38 | 2.50 | 2.47 | — |
No meaningful overall gain. Cross-push 2.50 -> 2.47%; within-push 2.38 -> 2.47%. Both inside the error bar on a CoV estimate at n=15 (approx +/-0.45pp at 1 sigma). Cost is +70% runtime.
4. Drop reddit-load link
Removed the reddit measure block. Remaining: 4 load + 3 subnav = 7 subtests.
| Metric | Fenix·fission | Fenix·nofis | GV·fission | GV·nofis |
|---|---|---|---|---|
| OVERALL | 79.25 (2.50%) | 80.68 (1.69%) | 79.22 (2.34%) | 82.64 (2.51%) |
| amazon-load | 50.87 (4.4%) | 51.51 (4.0%) | 51.97 (6.0%) | 53.61 (4.6%) |
| bbc-load | 38.26 (3.2%) | 38.99 (5.5%) | 37.10 (4.1%) | 40.10 (3.6%) |
| duckduckgo-load | 57.79 (4.1%) | 58.63 (2.6%) | 56.32 (2.4%) | 59.85 (4.4%) |
| wikipedia-load | 120.91 (4.0%) | 122.75 (5.0%) | 102.14 (5.4%) | 125.56 (7.6%) |
| amazon-subnav | 99.27 (4.8%) | 101.36 (3.8%) | 129.87 (3.6%) | 117.41 (2.7%) |
| duckduckgo-subnav | 114.22 (5.1%) | 118.24 (4.0%) | 108.89 (5.2%) | 107.56 (6.4%) |
| wikipedia-subnav | 127.71 (2.4%) | 129.00 (3.4%) | 125.43 (2.7%) | 129.79 (2.9%) |
Overall CoV moved only 2.56 -> 2.50% (Fenix) — the overall was already floor-limited, so reddit's high CoV was diluted in the geomean. reddit-load itself was 7.5-7.8% CoV on this pool (the noisiest subtest, and ~7% of that irreducible), so the win is in per-subtest alerting rather than the composite.
Conclusion
Dropped reddit-load (we can iterate on this in a follow-up for something better).
5 -> 9 browser cycles didn't provide meaningful improvement and runtime nearly doubled, so keeping 5 makes sense.
For reference, used this query to choose top sites. Some didn't work out (like facebook and google) and coincidentally ended up with the same sitelist as desktop!
|
|
Assignee | |
Updated•3 months ago
|
|
|
Assignee | |
Comment 4•1 month ago
|
||
|
|
Assignee | |
Comment 5•1 month ago
|
||
https://bugzilla.mozilla.org/show_bug.cgi?id=2058679 will likely land earlier, then we can probably run nav bench on fenix NaR as well
|
|
Assignee | |
Comment 6•18 days ago
•
|
||
also waiting on try runs to see if if direct mode will work too and enabling gecko profile,
|
|
Assignee | |
Comment 7•18 days ago
|
||
also i might as well add annotated videos to mobile
we currently don't do that for mobile tp6 due to some issues in the past https://bugzilla.mozilla.org/show_bug.cgi?id=2011884
but we can probably apply it to just nav bench mobile
|
|
Assignee | |
Comment 8•16 days ago
|
||
gecko profiles look good
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=59e25e2a959b4c19dcf5dae5a7359c459a877fce&searchStr=prof&selectedTaskRun=fowlt8G3SRCl_uLShwOpLw.0
large queue in android tasks, just waiting on direct mode + annotated videos run....
|
|
Assignee | |
Comment 9•16 days ago
|
||
direct mode isn't working https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=8dada0dba1ab05e2d89e58aee00e52af41ed05e3
i'll revert to proxy mode for the time being and give direct mode a closer look in a follow up
|
||
Comment 10•16 days ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #9)
direct mode isn't working https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=8dada0dba1ab05e2d89e58aee00e52af41ed05e3
i'll revert to proxy mode for the time being and give direct mode a closer look in a follow up
I can look into this, the Android direct mode issue. It was tested with Raptor TP6, but of course not nav-bench since we weren't there yet.
|
|
Assignee | |
Comment 11•16 days ago
|
||
awesome!
|
|
Assignee | |
Comment 12•16 days ago
•
|
||
while waiting for more try runs to finish... prelim look at the noise
for this push (59e25e2a959b)
nav-bench-overall across retriggers
| Variant | Jobs | Mean score | Std dev | CoV | Min | Max |
|---|---|---|---|---|---|---|
| shippable | 7 | 88.6 | 2.77 | 3.1% | 85.6 | 94.2 |
| NaR | 10 | 92.2 | 2.32 | 2.5% | 89.5 | 97.2 |
Per subtest score
"Across jobs" is the CoV of the per job value, "within job" is the median CoV of the 9 cycles inside a job.
| Subtest | shippable mean | shippable CoV (across jobs) | shippable CoV (within job, median) | NaR mean | NaR CoV (across jobs) | NaR CoV (within job, median) |
|---|---|---|---|---|---|---|
| amazon-nav-load | 56.3 | 4.8% | 8.2% | 58.6 | 3.8% | 6.8% |
| amazon-nav-subnav | 129.7 | 5.5% | 6.0% | 132.6 | 4.5% | 8.0% |
| bbc-nav-load | 40.7 | 5.8% | 5.9% | 42.5 | 5.9% | 5.6% |
| duckduckgo-nav-load | 66.8 | 3.0% | 4.3% | 71.0 | 2.9% | 3.7% |
| duckduckgo-nav-subnav | 128.7 | 6.3% | 9.0% | 137.1 | 4.4% | 7.8% |
| wikipedia-nav-load | 133.3 | 4.6% | 10.0% | 134.5 | 5.4% | 11.8% |
| wikipedia-nav-subnav | 126.1 | 1.7% | 3.7% | 131.7 | 2.6% | 4.8% |
Overall score is pretty stable at around 3% CoV. Naturally, i expect android to be noisier compared to desktop.
NaR scores ~4% higher than shippable, which is expected (found similar % difference for NaR sp3 fenix)
|
|
||
Comment 13•15 days ago
|
||
Looking good 👍.
This one-liner should fix the direct mode issues on Fenix: ensure gecko is not in proxy mode
Try push of your stack with the change
|
|
Assignee | |
Comment 14•15 days ago
|
||
woo nice!
|
|
Assignee | |
Comment 15•15 days ago
•
|
||
quick noise check:
| Config | Build | Jobs | Mean score | CoV across jobs | Median job time |
|---|---|---|---|---|---|
| 9 cycles, proxy | shippable | 17 | 89.9 | 2.7% | 59 min |
| NaR | 15 | 91.8 | 3.1% | 59 min | |
| 9 cycles, direct | shippable | 18 | 89.0 | 2.8% | 59 min |
| NaR | 17 | 90.8 | 3.3% | 59 min | |
| 6 cycles, proxy | shippable | 18 | 86.7 | 2.1% | 41 min |
| NaR | 17 | 91.3 | 3.2% | 40 min |
- overall score is still pretty good (2-3% COV)
- going from 9 -> 6 browser cycles doesn't seem to affect noise much
- Noisiest subtests are wikipedia load and bbc load (around 8 to 12% CoV between cycles in a job)
subtest breakdown:
"Across jobs" is how much the per job value moves between retriggers. "Within a job" is the median spread between the cycles inside a single job.
CoV across jobs:
| Subtest | 9 proxy ship | 9 proxy NaR | 9 direct ship | 9 direct NaR | 6 proxy ship | 6 proxy NaR |
|---|---|---|---|---|---|---|
| amazon-nav-load | 4.0% | 5.7% | 5.6% | 6.1% | 4.1% | 5.2% |
| amazon-nav-subnav | 4.1% | 5.7% | 4.6% | 3.5% | 3.9% | 4.5% |
| bbc-nav-load | 4.6% | 3.9% | 5.3% | 6.8% | 5.3% | 6.0% |
| duckduckgo-nav-load | 2.5% | 4.2% | 3.5% | 3.8% | 4.2% | 3.8% |
| duckduckgo-nav-subnav | 5.1% | 5.4% | 6.0% | 5.4% | 5.7% | 8.2% |
| wikipedia-nav-load | 5.9% | 5.1% | 7.4% | 5.3% | 4.1% | 5.4% |
| wikipedia-nav-subnav | 2.0% | 1.9% | 1.8% | 2.4% | 2.2% | 2.7% |
Median CoV within a job:
| Subtest | 9 proxy ship | 9 proxy NaR | 9 direct ship | 9 direct NaR | 6 proxy ship | 6 proxy NaR |
|---|---|---|---|---|---|---|
| amazon-nav-load | 4.6% | 6.9% | 7.6% | 9.5% | 7.2% | 6.1% |
| amazon-nav-subnav | 6.7% | 7.0% | 6.6% | 6.3% | 7.2% | 7.1% |
| bbc-nav-load | 7.3% | 6.8% | 11.2% | 7.9% | 6.3% | 6.2% |
| duckduckgo-nav-load | 3.8% | 4.5% | 4.4% | 4.4% | 4.1% | 3.6% |
| duckduckgo-nav-subnav | 6.5% | 8.2% | 6.7% | 7.1% | 7.9% | 5.1% |
| wikipedia-nav-load | 10.3% | 10.4% | 11.8% | 10.8% | 9.6% | 7.6% |
| wikipedia-nav-subnav | 4.7% | 4.8% | 4.7% | 4.4% | 4.4% | 4.6% |
Every subtest stays under about 8% between retriggers in all configs. wikipedia subnav is the steadiest by far. wikipedia load is the jumpiest between cycles (around 10%), but it averages out across a job.
Coming soon: direct mode with 4, 5 and 6 cycles, so we can pick the lowest cycle count that keeps noise reasonable
|
|
Assignee | |
Comment 16•15 days ago
|
||
noise update with everything ran so far
| Config | Build | Jobs | Mean score | CoV across jobs | Median job time |
|---|---|---|---|---|---|
| 9 cycles, proxy | shippable | 17 | 89.9 | 2.7% | 59 min |
| NaR | 15 | 91.8 | 3.1% | 59 min | |
| 6 cycles, proxy | shippable | 18 | 86.7 | 2.1% | 41 min |
| NaR | 17 | 91.3 | 3.2% | 40 min | |
| 9 cycles, direct | shippable | 18 | 89.0 | 2.8% | 59 min |
| NaR | 17 | 90.8 | 3.3% | 59 min | |
| 6 cycles, direct | shippable | 17 | 88.9 | 2.9% | 40 min |
| NaR | 19 | 90.2 | 3.2% | 40 min | |
| 5 cycles, direct | shippable | 17 | 87.9 | 3.3% | 33 min |
| NaR | 18 | 89.0 | 3.1% | 35 min | |
| 4 cycles, direct | shippable | 17 | 85.9 | 2.5% | 28 min |
| NaR | 19 | 92.4 | 3.2% | 27 min |
- Overall score sits around 2.5 to 3.3% CoV in every configuration, and cycle count barely moves it. (going from 9 down to 4 cycles doesn't seem to make any difference)
- most of the noise coming from the job itself rather than from not having enough cycles
- 4 cycles runs in about 28 min vs 59 min for 9, so roughly half the device time for the same stability
Subtest breakdown ( just direct mode since that is what we will use)
"Across jobs" is how much the per job value moves between retriggers. "Within a job" is the median spread between the cycles
CoV across jobs:
| Subtest | 9 ship | 9 NaR | 6 ship | 6 NaR | 5 ship | 5 NaR | 4 ship | 4 NaR |
|---|---|---|---|---|---|---|---|---|
| amazon-nav-load | 5.6% | 6.1% | 5.7% | 4.9% | 5.4% | 6.1% | 6.2% | 5.2% |
| amazon-nav-subnav | 4.6% | 3.5% | 5.7% | 5.1% | 5.5% | 6.1% | 2.9% | 6.0% |
| bbc-nav-load | 5.3% | 6.8% | 7.1% | 7.3% | 8.5% | 6.8% | 6.6% | 6.9% |
| duckduckgo-nav-load | 3.5% | 3.8% | 3.4% | 3.8% | 4.3% | 3.8% | 3.1% | 4.8% |
| duckduckgo-nav-subnav | 6.0% | 5.4% | 6.2% | 6.2% | 7.0% | 5.4% | 5.8% | 5.8% |
| wikipedia-nav-load | 7.4% | 5.3% | 5.5% | 5.9% | 6.4% | 4.6% | 4.9% | 7.2% |
| wikipedia-nav-subnav | 1.8% | 2.4% | 2.1% | 2.2% | 2.7% | 1.9% | 3.3% | 3.3% |
Median CoV within a job:
| Subtest | 9 ship | 9 NaR | 6 ship | 6 NaR | 5 ship | 5 NaR | 4 ship | 4 NaR |
|---|---|---|---|---|---|---|---|---|
| amazon-nav-load | 7.6% | 9.5% | 8.5% | 8.8% | 6.8% | 9.1% | 6.3% | 5.8% |
| amazon-nav-subnav | 6.6% | 6.3% | 5.7% | 5.9% | 5.8% | 6.0% | 7.1% | 6.1% |
| bbc-nav-load | 11.2% | 7.9% | 8.5% | 10.4% | 10.8% | 8.4% | 11.4% | 6.5% |
| duckduckgo-nav-load | 4.4% | 4.4% | 3.4% | 3.9% | 4.1% | 3.9% | 3.7% | 4.4% |
| duckduckgo-nav-subnav | 6.7% | 7.1% | 7.4% | 7.9% | 6.6% | 7.7% | 6.7% | 6.6% |
| wikipedia-nav-load | 11.8% | 10.8% | 9.7% | 9.9% | 10.1% | 9.9% | 10.6% | 9.8% |
| wikipedia-nav-subnav | 4.7% | 4.4% | 3.7% | 4.5% | 3.7% | 3.9% | 3.5% | 3.9% |
Subtests stay under about 8.5% between retriggers in every config. bbc load and wikipedia load are the jumpiest, wikipedia subnav is the steadiest.
this seems to suggest we could use direct mode with 4 or 5 cycles. (maybe even less?)
Maybe we do 5 just to be consistent with desktop?
|
|
||
Comment 17•11 days ago
|
||
(In reply to Kash Shampur [:kshampur] ⌚EST from comment #16)
...
this seems to suggest we could use direct mode with 4 or 5 cycles. (maybe even less?)
Maybe we do 5 just to be consistent with desktop?
Great work!
My vote is yes, ship it with 5 cycles like desktop.
|
||
Updated•11 days ago
|
|
||
Comment 18•8 days ago
|
||
|
||
Comment 19•8 days ago
|
||
| bugherder | ||
Description
•