1663138 - 6.53 - 12.17% perf_reftest (linux64-shippable, linux64-shippable-qr) regression on push 57b1de9d90dce7d2050d3894aa5b8b03f3e1460d (Mon August 31 2020)

Status: RESOLVED WONTFIX

(Firefox Build System :: Toolchains, defect)

Description

[Florin Strugariu [:Bebe]]

Perfherder has detected a talos performance regression from push 57b1de9d90dce7d2050d3894aa5b8b03f3e1460d. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

12% perf_reftest_singletons tiny-traversal-singleton.html linux64-shippable-qr opt e10s stylo 708.60 -> 794.84
11% perf_reftest_singletons tiny-traversal-singleton.html linux64-shippable opt e10s stylo 668.35 -> 742.00
8% perf_reftest_singletons coalesce-2.html linux64-shippable-qr opt e10s stylo 132.12 -> 142.96
7% perf_reftest_singletons coalesce-1.html linux64-shippable-qr opt e10s stylo 162.50 -> 173.88
7% perf_reftest_singletons parent-basic-singleton.html linux64-shippable-qr opt e10s stylo 82.26 -> 87.63

Improvements:

24% basic_compositor_video linux64-shippable-qr opt e10s stylo 3.66 -> 2.80
21% basic_compositor_video linux64-qr opt e10s stylo 3.55 -> 2.81

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Comment 1

[Florin Strugariu [:Bebe]]

One of the bugs in this patch generated a regression can you take a look

Comment 2

[(Away)]

Please let me know when 100% of the alerts for this landing have been processed and no more are expected. It is unlikely that we'll be able to mitigate individual test cases so we'll need to look at the full picture and decide whether to accept. Having regressions come in as separate bugs over time makes it difficult to make a proper assessment, so we'll need to wait for the complete data.

Comment 3

[(Away)]

Any news on this? Are there still more pending data points to be processed?

Comment 4

[Florin Strugariu [:Bebe]]

I think no other alerts will be generated

Comment 5

[(Away)]

That's great to hear. In that case this is less fallout than I was expecting. Given that these are some isolated subtests, and we got some improvements to balance the regressions, I think this is an acceptable amount of churn for something as intrusive as a compiler change. I'm inclined to mark this as wontfix unless anyone wants to push for otherwise.

Comment 6

[Florin Strugariu [:Bebe]]

I talked to soon :D we found some small regressions on raptor also
== Change summary for alert #26953 (as of Thu, 17 Sep 2020 05:28:07 GMT) ==

Regressions:

5% raptor-tp6-yahoo-mail-firefox-cold loadtime windows10-64-shippable opt 1,114.31 -> 1,165.00
3% raptor-stylebench-firefox windows10-64-shippable-qr opt webrender 46.54 -> 44.96
2% raptor-stylebench-firefox windows10-64-shippable-qr opt webrender 46.18 -> 45.08

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26953

Comment 7

[Julien Cristau [:jcristau]]

Should we close this bug as wontfix at this point?

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Julien Cristau [:jcristau] from comment #7)

Should we close this bug as wontfix at this point?

It sounds like it, although the 5% cold page load regression for Yahoo! Mail on Windows 10 is concerning. :bebe looking at the graph I suspect there may be an earlier cuplrit for this alert. Could you investigate?

Comment 9

[Florin Strugariu [:Bebe]]

reassigned the yahoo alert to https://treeherder.mozilla.org/perf.html#/alerts?id=26885