Closed Bug 1898413 Opened 2 months ago Closed 2 months ago

3.12 - 3.09% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync / speedometer EmberJS-Debug-TodoMVC/CompletingAllItems (OSX) regression on Thu May 16 2024

Categories

(Firefox Build System :: Toolchains, defect)

defect

Tracking

(firefox-esr115 unaffected, firefox126 unaffected, firefox127 unaffected, firefox128 fixed)

RESOLVED FIXED
128 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox126 --- unaffected
firefox127 --- unaffected
firefox128 --- fixed

People

(Reporter: afinder, Assigned: glandium)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Attachments

(3 files)

Perfherder has detected a browsertime performance regression from push c6dd0210975b42f6a4d9e08b8047cf1e4f13ffe5. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
3% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync macosx1400-64-shippable-qr fission webrender 64.15 -> 66.15 Before/After
3% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems macosx1400-64-shippable-qr fission webrender 65.26 -> 67.28 Before/After

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
4% speedometer Vanilla-ES2015-TodoMVC/CompletingAllItems/Sync macosx1015-64-shippable-qr fission webrender 14.62 -> 14.00 Before/After
4% speedometer Vanilla-ES2015-Babel-Webpack-TodoMVC/CompletingAllItems/Sync macosx1015-64-shippable-qr fission webrender 14.59 -> 13.99 Before/After
4% speedometer Vanilla-ES2015-TodoMVC/CompletingAllItems macosx1015-64-shippable-qr fission webrender 16.52 -> 15.85 Before/After
3% webaudio windows10-64-shippable-qr fission webrender 81.21 -> 78.42 Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 402

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(mh+mozilla)

Set release status flags based on info from the regressing bug 1878764

I profiled Nightly builds from 2024-05-15 and 2024-05-17 locally on an M1Max Macbook Pro.
2024-05-15: https://share.firefox.dev/456incU
2024-05-17: https://share.firefox.dev/4aKVj4I

I don't see anything explaining a 3% difference. But I do see a 1.3% difference in array_join.

The profiles were captured on https://extra--speedometer-preview.netlify.app/?suite=EmberJS-Debug-TodoMVC&iterationCount=100
See the Profile Info dropdown for profiler settings - I used an interval of 0.4ms and generally chose profiler settings with low overhead.

The assembly looks quite similar and yet the instruction sample counts are quite different. I don't have an explanation at the moment.

Flags: needinfo?(mh+mozilla)
Regressed by: clang-18
No longer regressed by: rustc-1.78

This has the potential to regress other things, and if that happens,
we'll balance the pros and cons. But first, let's attempt this.

Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/2d836066995d
Revert clang 18 change that causes regressions in some speedometer tests. r=firefox-build-system-reviewers,sergesanspaille
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → 128 Branch

(In reply to Pulsebot from comment #7)

Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/2d836066995d
Revert clang 18 change that causes regressions in some speedometer tests.
r=firefox-build-system-reviewers,sergesanspaille

Perfherder has detected a browsertime performance change from push 2d836066995d9c7cdb5c191401fa8bd393bb0eab.

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
3% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems macosx1400-64-shippable-qr fission webrender 67.05 -> 64.83 Before/After
3% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync macosx1400-64-shippable-qr fission webrender 65.90 -> 63.71 Before/After
2% speedometer EmberJS-Debug-TodoMVC macosx1400-64-shippable-qr fission webrender 213.45 -> 209.09 Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 647

For more information on performance sheriffing please see our FAQ.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: