1541448 - Check if Speedometer tests have to run with 5 page cycles

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

On bug 1460741 the Speedometer tests have been added to Raptor about 10 months ago.

While checking how this test suite works I noticed that it runs 10 iterations itself internally. You can see it when clicking the details button once the results are shown.

With Raptor we run this suite with 5 page_cycles which actually results in 50 runs.

Not sure if I miss something but is that behavior wanted?

Comment 1

[Robert Wood [:rwood]]

We ported the speedometer (and other benchmarks) from Talos into Raptor. In Talos speedometer was also running 5 page-cycles [0]. I'm guessing it was determined due to data noise that 5 page-cycles was what was ideally required to reduce noise and get the best automated regression detection.

[0] https://searchfox.org/mozilla-central/rev/14dc5b7d8a6da1854b2f9f33f1da77a97368cd54/testing/talos/talos/test.py#837

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

if we determine the results are reliable with fewer iterations, then lets do that, it would only save time. Keep in mind this runs on linux, osx, windows, android

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Attachment: Screenshot_2019-04-04 Speedometer 1 0.png

Attached a screenshot from an execution triggered manually by loading the benchmark via a file:// URL.

By using index.html?raptor which is in use by Raptor, the tests run in about 45s whereby without the paramter it takes 1:30min. So there is something special when the benchmark is run under Raptor, beside removing the surrounding UI and final results.

Joel, maybe you know what that does?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

your best bet is to look at the source code, here is where we get speedometer from:
https://searchfox.org/mozilla-central/search?q=raptor&path=third_party%2Fwebkit%2FPerformanceTests%2F

I included the root directory as there might be shared files with other tests.

Alternatively if you want to look at noise, you can pull the raw data from activedata and look at the score for each of the 5 runs over time to see the noise.

Another choice is to push to try with default and with cycles=1, then use compare view to see what differences there are.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #4)

Another choice is to push to try with default and with cycles=1, then use compare view to see what differences there are.

Alex, do this, please.

Comment 6

[Alex Finder|PTO, returning Feb 17th]

Here we can view the comparison between page cycle 1 (revision e15225132a89) and page cycle 5 (revision 88421ee7352d) compared to autoland for platforms Linux1804-64-shippable-qr, Macosx1015-64-shippable-qr and windows10-64-shippable-qr. There does not seem to be much difference in noise between the two values. The only platform that shows less noise for page cycle 5 is windows10-64-shippable-qr. So, I suggest we leave it set to 5 for the moment.

Comment 7

[Alex Finder|PTO, returning Feb 17th]

Looking at the results for the android platforms, these do not seem to be very consistent. android-hw-a51-11-0-arm7-qr results seem to be more noisy for page cycle 5, although with android-hw-g5-7-0-arm7-qr, android-hw-p2-8-0-android-aarch64-qr and android-hw-p2-8-0-arm7-qr we have more noise for page cycle 1.

Comment 8

[Alex Finder|PTO, returning Feb 17th]

Updated the comparison after landing the "Update speedometer 2.1" patch between page cycle 1 and page cycle 5 compared to autoland for platforms Linux1804-64-shippable-qr which shows better/higher results for page cycle 1, Macosx1015-64-shippable-qr showing less noise for page cycle 5, and windows10-64-shippable-qr with higher/better results for page cycle 5. For android we have the following results: android-hw-a51-11-0-arm7-qr, android-hw-g5-7-0-arm7-qr, android-hw-p2-8-0-android-aarch64-qr and android-hw-p2-8-0-arm7-qr. These seem to show less noise and higher/better results for 5 page cycles on android-hw-g5-7-0-arm7-qr.

Comment 9

[Alex Finder|PTO, returning Feb 17th]

Based on the results, I'll update the page cycle value to 5 for the speedometer-mobile.ini and submit the patch.

Comment 10

[Alex Finder|PTO, returning Feb 17th]

Attachment: Bug 1541448 Check if Speedometer tests have to run with 5 page cycles r=#perftest

Comment 11

[Alex Finder|PTO, returning Feb 17th]

Attachment: Bug 1541448 Check if Speedometer tests have to run with 5 page cycles r=#perftest

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c1b13c1d115
Check if Speedometer tests have to run with 5 page cycles r=perftest-reviewers,AlexandruIonescu

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/1c1b13c1d115