1931745 - Ares6 and Sunspider scores on Linux-js-bench are 5 points lower than the same test on Browsertime

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect, P3)

Description

[Mayank Bansal]

Ares6
Js-bench: Values are around 50ms
Browsertime: Values are around 45ms

Sunspider
js-bench around 200ms
Browsertime around 187ms

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:kshampur, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

js-bench and raptor-browsertime are different enough harnesses, and the target applications are also different, that I don't think it makes a lot of sense to compare the results. As long as the results are consistent and able to detect changes, I think we should close this as wontfix. @sparky what do you think?

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

changing from talos -> raptor since these are the browsertime benchmarks mentioned here in comparison to js bench

Comment 4

[Greg Mierzwinski [:sparky]]

Actually, it looks like the sunspider one diverged in the last year. See here for January, on around the 14th there was an improvement for browsertime but not for js-bench: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,2345107,1,11&series=autoland,3940287,1,13&timerange=31536000&zoom=1703802640688,1706993297729,154.11167516770706,253.2515444487528

There's a similar drop on the same date/push in the Ares6 test on browsertime which isn't found in the js-bench variant: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&replicates=0&series=autoland,2345224,1,11&series=autoland,3915286,1,13&series=mozilla-central,3406048,1,13&timerange=31536000&zoom=1704500678449,1706213891715,42.654519102795405,54.41906927580578

For Ares6 though, the divergence between the harnesses already existed.

These were caused by one of the patches in this mozilla-central push: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=53d4d65c33f1a556689dd9de738b173142e52749&tochange=23a77d3c25d74ed46ebb77f47fda83c7b6dcdb45

Mayank, do you see any patches there that may have caused this improvement?

Comment 5

[Mayank Bansal]

4 patches in this range are relevant :

Bug 1874376 / 1869726 and Bug 1874256 : These improve JS performance. But I dont know why these would only impact Browsertime, and not js-bench.
Bug 1649696: This improves all Linux performance, so is potentially a contender too.

---

My first attempt was to look at what Bug 1874376 / Bug 1869726 and Bug 1874256 were improving. I compared the improvements reported in those bugs with Sunspider and Ares6 benchmarks. That didnt go too far, as none of sunspider or Ares6 improved by those 3 bugs.

My second attempt was to look at the sub-benchmarks of Sunspider and Ares6, see which of them improved on 14Jan2024 and look for areas where they again improved throughout this year, in the hope that this will give me some insight on what type of JS perf bugs improves those sub-tests, and see if there is any similarity between those JS perf bugs and our 4 contenders.
This again didnt go too far as not many sub-tests improved, and those which did improve didnt improve much throughout this year.

The third attempt was to compare Linux with MAc and Windows for the Jan2024 period. See the graphs below :
Windows Ares6
Windows sunspider
MacOS Ares6
MacOS Sunspider
That gives some clues.... None of Mac and Windows benchmarks improved. Bug 1874376 / 1869726 and Bug 1874256 are engine improvements so should improve all the three platforms.
Therefore the conclusion is that the improvements were Linux specific. Out of our 4 contenders, only Bug 1649696 fits that bill.

So my final conclusion is that Bug 1649696 is the regressor in that range, and for some reason it only improved Browsertime.
(Wild theory : comment in https://phabricator.services.mozilla.com/D198217 says this :
Note that, if a seccomp policy is applied by some external sandbox which doesn't use this opt-out (e.g., if using Snap or Flatpak), and the kernel is in that range, these Spectre mitigations will still be applied and we can't turn them off.
Is js-bench applying its own seccomp policy?)

This is all speculation, of course.

Comment 6

[Greg Mierzwinski [:sparky]]

That makes sense, I agree with bug 1649696 being the most likely culprit. I was looking around at the harness code and the most likely cause for this difference is that the js-bench harness executes the benchmark through a js shell, but browsertime is executing it through the browser with the benchmark being served through a webserver locally.

I wonder if someone from the JS team could help. Not sure who builds the js shell there though. :jandem, would you have any thoughts on this issue? I'm wondering that if bug 1649696 was the culprit, why would the jsshell not be affected by it, but the browser is.

Comment 7

[Jan de Mooij [:jandem]]

(In reply to Greg Mierzwinski [:sparky] from comment #6)

I wonder if someone from the JS team could help. Not sure who builds the js shell there though. :jandem, would you have any thoughts on this issue? I'm wondering that if bug 1649696 was the culprit, why would the jsshell not be affected by it, but the browser is.

The JS shell doesn't have any (seccomp) sandboxing, so it makes sense that bug 1649696 only affected the browser. Bug 1649109 also confirms that this only affected the browser and not the JS shell.

Comment 8

[Greg Mierzwinski [:sparky]]

Ok, thanks! I think we can close this as a wontfix now. There's nothing else we can do for the divergence in sunspider, and the ares6 divergence can't be investigated further.