1373723 - Speedometer async benchmarks performance is very flaky

Status: RESOLVED WONTFIX

(Core :: DOM: Core & HTML, defect, P2)

Description

[Benjamin Bouvier [:bbouvier] (inactive)]

When looking at any of the Speedometer2 tests that contain "async" in their name:
https://arewefastyet.com/#machine=35&view=single&suite=speedometer-misc
(win64)
https://arewefastyet.com/#machine=17&view=single&suite=speedometer-misc
(win32)
https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc

It appears that the performance on Firefox is very flaky and has a lot of variance, while the performance on Chromium is very stable. This suggests a bug on our side. I haven't profiler these tests, nor investigated: first ideas which come to mind are GC or the event loop handling.

Comment 1

[Jan de Mooij [:jandem]]

This is basically bug 1366803. A number of bugs were fixed to run stuff during idle time, but there's more to be done here.

Comment 2

[Olli Pettay [:smaug][[e-mail]]]

FWIW, we don't have enough idle time to run GC/CC during idle period when executing Speedometer.
I'm planning to try to increase NS_INTERSLICE_GC_DELAY and kICCIntersliceDelay to see if that would give a bit better chance to use idle time, but I doubt that will help much.

Comment 3

[Jan de Mooij [:jandem]]

I hope nursery allocated strings and the GC scheduler will help.. Do you know what's triggering the GCs on Speedometer?

Comment 4

[Olli Pettay [:smaug][[e-mail]]]

mostly GC alloc, and also some CC_WAITING

Comment 5

[(no longer active)]

Should we dupe it against bug 1366803?

Comment 6

[(no longer active)]

Attachment: Mark the async test times as a UserTiming region in the profiler UI

I have been using this patch to mark the async test periods as a UserTiming area in the profiler UI.  It is helpful for profiling this period exclusively.

Comment 7

[(no longer active)]

Markus noticed that the backbone tests seem to run into issues where a timer that the test schedules may end up running at within the async period: http://bit.ly/2tOFUwU

The AWFY graphs seem to suggest that this test is a source of noise in the benchmark for us.  This might explain why.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

(In reply to :Ehsan Akhgari (needinfo please, extremely long backlog) from comment #7)
> Markus noticed that the backbone tests seem to run into issues where a timer
> that the test schedules may end up running at within the async period:
> http://bit.ly/2tOFUwU
> 
> The AWFY graphs seem to suggest that this test is a source of noise in the
> benchmark for us.  This might explain why.

Note, chrome current clamps all setTimeout() calls to a minimum of 1ms.  So a setTimeout(f, 0) in chrome will run later than for us.  I'm not sure we want to do that, but perhaps it helps them here.

Comment 9

[Vicky Chin [:vchin]]

Tagging this with [qf] as async tests on speedometer still have a lot of variance:
https://arewefastyet.com/#machine=35&view=breakdown&suite=speedometer-misc

Comment 10

[Andrew Creskey [:acreskey]]

I ran these tests locally through the raptor framework, and yes, the async tests show a relative standard deviation that is many times that of the sync tests.

As a quick experiment, I implemented a 1ms clamp on the timeout intervals (https://phabricator.services.mozilla.com/D10946).
I believe this is correctly implemented.

However when running with this change I did not see a significant reduction in variance on the async tests.

Comment 11

[Olli Pettay [:smaug][[e-mail]]]

sp2 runner is inherently just buggy. It may contain rAF/painting or may not. sp3 has better runner.