1339557 - (Speedometer_V2) [Meta] Quantum Release Criteria: Figure out why we are so slow on Speedometer v2

Status: NEW

(Core :: JavaScript Engine, defect, P2)

Description

[(no longer active)]

See <https://twitter.com/bz_moz/status/831585613258379265>

We have looked at this before a bit, see bug 1245279 and bug 1022988.  We need to profile again and find out why we are so slow.  This is important since this test executes a bunch of popular frameworks.

Andrew, can we find an owner for this please?

Comment 1

[Andrew Overholt [:overholt]]

Let me work on it.

Comment 2

[(no longer active)]

I spent a bit of time looking at some random profiles from a long Speedometer run and all I can see is JS.  Perhaps someone on the JS team would be a better candidate.

Comment 3

[David Ross]

Marked improvement overnight after fix via bug: 1339535

My fiber today is shocking for some reason, but still seeing a ~35% speed improvement. Testing in another location in around an hour.

Arch Linux rolling release. Nightly 2017-02-28 vs 2017-02-27. Dell Latitude E4300 16Gb RAM.

Comment 4

[David Ross]

Despite improvements to Nightly, which I've experienced on my laptop since Feb 27, testing is now shockingly slow on a Galaxy Tab A (9.7) running Fx55. Speedometer test been running for 45 mins and not yet half way :(

Comment 5

[Ben Kelly [:bkelly, not reviewing]]

Can we put a link to the v2 speedometer in this bug?  Not sure where it lives.

Comment 6

[Andrew Overholt [:overholt]]

I added it to the URL field of this bug.

Comment 7

[TMart]

According to bug 1347525 slow InnerHTML is the main reason for Firefox's slowness in Speedometer. This was resolved, but Speedometer is still at best only half as fast as in Chrome for me. 

With an i7 6700k I get 199 runs/minute in Chrome, 98 runs/minute in Firefox. This scales down with CPU power, on an i5 6200U I get 42 runs/minute with Firefox, 88 with Chrome.

How relevant is Speedometer for Gecko, anyway? V8 describes it as a great tool for measuring real world performance, however in bug 1347525 comment #3 seems to suggest otherwise.

Comment 8

[Jan de Mooij [:jandem]]

(In reply to TMart from comment #7)
> According to bug 1347525 slow InnerHTML is the main reason for Firefox's
> slowness in Speedometer.

I never said that. In that bug I was talking about one of the subtests, but there are a lot of these and we knew fixing that bug would not magically fix everything.

> With an i7 6700k I get 199 runs/minute in Chrome, 98 runs/minute in Firefox.
> This scales down with CPU power, on an i5 6200U I get 42 runs/minute with
> Firefox, 88 with Chrome.

This might be better with Speedometer v2.

> How relevant is Speedometer for Gecko, anyway? V8 describes it as a great
> tool for measuring real world performance, however in bug 1347525 comment #3
> seems to suggest otherwise.

There are a bunch of silly things for sure. Overall it's not an unreasonable benchmark though and so far almost all performance issues we found/fixed will help real-world websites too.

Comment 10

[Armen [:armenzg]]

The machine that runs speedometer got this morning a change that will make the score of speedometer improve (bug 1373396).

I don't know how it will affect the score of Chrome but we will know later today. AWFY is currently running:
https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc

Also note that we currently run non-PGO builds for speedometer which could mean that our score could even be a bit higher.

Comment 11

[Ting-Yu Chou [:ting] (away)]

I see PollWrapper() in profiles, has anyone tried to understand why? Like is it the test designed to short break between runs, or is it firefox waiting on something?

Comment 12

[(no longer active)]

(In reply to Ting-Yu Chou [:ting] from comment #11)
> I see PollWrapper() in profiles, has anyone tried to understand why? Like is
> it the test designed to short break between runs, or is it firefox waiting
> on something?

The test schedules a lot of asychronous work, such as loading web pages from the network, but it doesn't measure the entire time that it runs.  See this function for what the benchmark really measures: https://github.com/WebKit/webkit/blob/057552ca85b3efd9c97d13a0f044abd0daba7526/PerformanceTests/Speedometer/resources/benchmark-runner.js#L121

Comment 13

[Robin Steuber (they/them) [:bytesized]]

The previous Speedtest v2 URL showed a "Not Found" page. I have been told that this is the correct URL to use (despite a page title of "Speedtest 1.0", which I'm told is incorrect).

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Steven, I guess we can close this bug, right?

Comment 15

[Steven DeTar [:sdetar]]

Ted, do you think we can close this bug now?

Comment 16

[Olli Pettay [:smaug][[e-mail]]]

We are still significantly slower on Speedometer.

Comment 17

[Ted Campbell [:tcampbell]]

This is Jan's call, but as Olli points out, we still have a remaining perf difference.

Comment 18

[Jan de Mooij [:jandem]]

Yeah let's keep this open for now. Even though we're not as focused on speedometer as during QF we're still slower.