1898413 - 3.12 - 3.09% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync / speedometer EmberJS-Debug-TodoMVC/CompletingAllItems (OSX) regression on Thu May 16 2024

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Alex Finder]

Perfherder has detected a browsertime performance regression from push c6dd0210975b42f6a4d9e08b8047cf1e4f13ffe5. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
3%	speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync	macosx1400-64-shippable-qr	fission webrender	64.15 -> 66.15	Before/After
3%	speedometer EmberJS-Debug-TodoMVC/CompletingAllItems	macosx1400-64-shippable-qr	fission webrender	65.26 -> 67.28	Before/After

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
4%	speedometer Vanilla-ES2015-TodoMVC/CompletingAllItems/Sync	macosx1015-64-shippable-qr	fission webrender	14.62 -> 14.00	Before/After
4%	speedometer Vanilla-ES2015-Babel-Webpack-TodoMVC/CompletingAllItems/Sync	macosx1015-64-shippable-qr	fission webrender	14.59 -> 13.99	Before/After
4%	speedometer Vanilla-ES2015-TodoMVC/CompletingAllItems	macosx1015-64-shippable-qr	fission webrender	16.52 -> 15.85	Before/After
3%	webaudio	windows10-64-shippable-qr	fission webrender	81.21 -> 78.42	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 402

For more information on performance sheriffing please see our FAQ.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1878764

Comment 2

[Markus Stange [:mstange]]

I profiled Nightly builds from 2024-05-15 and 2024-05-17 locally on an M1Max Macbook Pro.
2024-05-15: https://share.firefox.dev/456incU
2024-05-17: https://share.firefox.dev/4aKVj4I

I don't see anything explaining a 3% difference. But I do see a 1.3% difference in array_join.

The profiles were captured on https://extra--speedometer-preview.netlify.app/?suite=EmberJS-Debug-TodoMVC&iterationCount=100
See the Profile Info dropdown for profiler settings - I used an interval of 0.4ms and generally chose profiler settings with low overhead.

Comment 3

[Markus Stange [:mstange]]

Attachment: 2025-05-15 build, assembly view of array_join

Comment 4

[Markus Stange [:mstange]]

Attachment: 2025-05-17 build, assembly view of array_join

Comment 5

[Markus Stange [:mstange]]

The assembly looks quite similar and yet the instruction sample counts are quite different. I don't have an explanation at the moment.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1898413 - Revert clang 18 change that causes regressions in some speedometer tests.

This has the potential to regress other things, and if that happens,
we'll balance the pros and cons. But first, let's attempt this.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2d836066995d
Revert clang 18 change that causes regressions in some speedometer tests. r=firefox-build-system-reviewers,sergesanspaille

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/2d836066995d

Comment 9

[Alex Finder]

(In reply to Pulsebot from comment #7)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2d836066995d
Revert clang 18 change that causes regressions in some speedometer tests.
r=firefox-build-system-reviewers,sergesanspaille

Perfherder has detected a browsertime performance change from push 2d836066995d9c7cdb5c191401fa8bd393bb0eab.

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
3%	speedometer EmberJS-Debug-TodoMVC/CompletingAllItems	macosx1400-64-shippable-qr	fission webrender	67.05 -> 64.83	Before/After
3%	speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync	macosx1400-64-shippable-qr	fission webrender	65.90 -> 63.71	Before/After
2%	speedometer EmberJS-Debug-TodoMVC	macosx1400-64-shippable-qr	fission webrender	213.45 -> 209.09	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 647

For more information on performance sheriffing please see our FAQ.