1503559 - Experiment: train PGO build with full websites, evaluate performance with Visual Metrics

Status: NEW

(Core :: Performance, enhancement, P3)

Description

[Andrew Creskey [:acreskey]]

Currently the PGO builds use the following set of tests and pages to train the optimizer:
https://searchfox.org/mozilla-central/source/build/pgo/index.html

These tests may or may not exercise sufficient codepaths to optimize for page load.
This bug is to run an experiment where the PGO is trained on live websites. For example, the sites in the tp6-p1 corpus.
Whether or not this produces a better optimization can be tested by running the a visual metrics comparison with Browsertime.

Comment 1

[Andrew Creskey [:acreskey]]

Blocks: https://bugzilla.mozilla.org/show_bug.cgi?id=1505256

Comment 2

[(no longer active)]

FWIW (sorry if I'm repeating something you already know) in Firefox 63 we switched from MSVC to clang-cl on Windows.  Before that for many years we didn't change the PGO training set all that much mostly out of the fear of triggering new PGO crashes caused by MSVC bugs.  Now we don't have that limitation any more \o/.

Bug 1356652 was the last major change to our PGO training set where we imported the Speedometer benchmark.  Besides that, the rest of what we run in the PGO training set hardly tests anything interesting in the browser.  There is sunspider and a couple of other pages that test a few DOM APIs and that's about it.

I'm very excited to see what improvements we can unlock by including better pagesets in our PGO training set.  Thanks a lot for taking on this challenge.

Comment 3

[Andrew Creskey [:acreskey]]

Hi Barret - we're thinking now that this would be better done with Visual Metrics.
Are you interested in trying this again?
I can run the visual metrics if you can supply the binaries...

Comment 4

[Andrew Creskey [:acreskey]]

Updated: a simpler strategy of training against live sites and comparing performance with the more meaningful visual metrics

Comment 5

[Barret Rennie [:barret] (they/them)]

Yeah I know the incantations to make PGO point at different sites, but I am busy probably for the rest of the quarter. I can take a look at this then.

Comment 6

[Andrew Creskey [:acreskey]]

I did a quick test where I saved a few sites (facebook/google/bing) to disk and ran the android PGO training run on them.

Maybe some modest ~3% improvements on facebook in this compare
and also here with an -O2 optimized binary.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=7482c843963313d2cd4104545cdb0f466121ab14&newProject=try&newRevision=44752d230a0c19256fe0f7b136cef4b37a852f56&framework=10