Open Bug 1373723 Opened 2 years ago Updated Last month

Speedometer async benchmarks performance is very flaky

Categories

(Core :: DOM: Core & HTML, defect, P2)

defect

Tracking

()

People

(Reporter: bbouvier, Unassigned)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [qf:p2:responsiveness])

Attachments

(1 file)

When looking at any of the Speedometer2 tests that contain "async" in their name:
https://arewefastyet.com/#machine=35&view=single&suite=speedometer-misc
(win64)
https://arewefastyet.com/#machine=17&view=single&suite=speedometer-misc
(win32)
https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc

It appears that the performance on Firefox is very flaky and has a lot of variance, while the performance on Chromium is very stable. This suggests a bug on our side. I haven't profiler these tests, nor investigated: first ideas which come to mind are GC or the event loop handling.
This is basically bug 1366803. A number of bugs were fixed to run stuff during idle time, but there's more to be done here.
FWIW, we don't have enough idle time to run GC/CC during idle period when executing Speedometer.
I'm planning to try to increase NS_INTERSLICE_GC_DELAY and kICCIntersliceDelay to see if that would give a bit better chance to use idle time, but I doubt that will help much.
I hope nursery allocated strings and the GC scheduler will help.. Do you know what's triggering the GCs on Speedometer?
mostly GC alloc, and also some CC_WAITING
Should we dupe it against bug 1366803?
I have been using this patch to mark the async test periods as a UserTiming area in the profiler UI.  It is helpful for profiling this period exclusively.
Priority: -- → P2
Markus noticed that the backbone tests seem to run into issues where a timer that the test schedules may end up running at within the async period: http://bit.ly/2tOFUwU

The AWFY graphs seem to suggest that this test is a source of noise in the benchmark for us.  This might explain why.
(In reply to :Ehsan Akhgari (needinfo please, extremely long backlog) from comment #7)
> Markus noticed that the backbone tests seem to run into issues where a timer
> that the test schedules may end up running at within the async period:
> http://bit.ly/2tOFUwU
> 
> The AWFY graphs seem to suggest that this test is a source of noise in the
> benchmark for us.  This might explain why.

Note, chrome current clamps all setTimeout() calls to a minimum of 1ms.  So a setTimeout(f, 0) in chrome will run later than for us.  I'm not sure we want to do that, but perhaps it helps them here.
Tagging this with [qf] as async tests on speedometer still have a lot of variance:
https://arewefastyet.com/#machine=35&view=breakdown&suite=speedometer-misc
Whiteboard: [qf]
Whiteboard: [qf] → [qf:p1:f67]
I ran these tests locally through the raptor framework, and yes, the async tests show a relative standard deviation that is many times that of the sync tests.

As a quick experiment, I implemented a 1ms clamp on the timeout intervals (https://phabricator.services.mozilla.com/D10946).
I believe this is correctly implemented.

However when running with this change I did not see a significant reduction in variance on the async tests.
Whiteboard: [qf:p1:f67] → [qf:p2:responsiveness]
Component: DOM → DOM: Core & HTML
Blocks: 1552348
You need to log in before you can comment on or make changes to this bug.