1867581 - When run through profileserver.py, speedometer3 hangs at 57/610 test case and hits 404

Status: RESOLVED FIXED

(Core :: Performance, defect)

Description

Since bug 1862309 , speedometer3 is part of the PGO training set, and it has already improved the speedometer3 score to do so.

However, when running the training, the test actually hangs at 56/610, hits a 404 and then timeouts, moving to next test. It would be great to have it run until the end.

Steps to reproduce

$ mach build
$ mach python ./build/pgo/profileserver.py -b $PWD/obj-x86_64-pc-linux-gnu/dist/bin/firefox-bin

And keep an eye on the window that pops up.

Comment 1

[:glob ✱]

I see a 404 on macOS also, although interestingly I see it at 57/610 (React-Stockcharts-SVG).

The command line for macOS is slightly different (use the binary under dist/Nightly.app not dist/bin):

$ mach python ./build/pgo/profileserver.py -b $PWD/obj-aarch64-apple-darwin23.1.0/dist/Nightly.app/Contents/MacOS/firefox-bin

Comment 2

[:glob ✱]

Attachment: Screenshot 2023-12-05 at 3.56.30 pm.png

Comment 3

[Marc Leclair [:mleclair]]

This is because some of the files ( I think some) has to be ran as root. I have a fix, but it is failing in ci

Comment 4

[Marc Leclair [:mleclair]]

Here's another try push with some logs and running all of the PGO tests.

Comment 5

[Marc Leclair [:mleclair]]

Attachment: Bug 1867581 - Fixed PGO training set not working for SP3 since SP3 wasn't running as root r=bas

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e7ca7b9c60d
Fixed PGO training set not working for SP3 since SP3 wasn't running as root r=bas

Comment 7

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/5e7ca7b9c60d

Comment 8

[[:sergesanspaille]]

Thanks a lot Marc for tracking this down. Did you run some benchmark to compare the impact of running the full speedometer3 suite on the quality of the profiled binary?

Comment 9

[Nicolas Chevobbe [:nchevobbe]]

Looks like this caused a few small improvements on DevTools tests, not sure why though

== Change summary for alert #40647 (as of Mon, 11 Dec 2023 23:46:24 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
3%	damp custom.jsdebugger.open-large-minified-file.DAMP	macosx1015-64-shippable-qr	e10s fission stylo webrender	1,333.64 -> 1,298.78
2%	damp source-map.eachMapping.DAMP	windows10-64-shippable-qr	e10s fission stylo webrender	257.47 -> 251.19
2%	damp custom.jsdebugger.pretty-print.reload-and-pause.DAMP	macosx1015-64-shippable-qr	e10s fission stylo webrender	7,930.54 -> 7,765.64
2%	damp custom.pretty-print.jsdebugger.reload.DAMP	macosx1015-64-shippable-qr	e10s fission stylo webrender	7,768.59 -> 7,608.71
2%	damp custom.jsdebugger.pretty-print.DAMP	macosx1015-64-shippable-qr	e10s fission stylo webrender	1,777.32 -> 1,741.13

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=40647