1575157 - 21.42 - 25.32% perf_reftest_singletons (linux64-shippable, linux64-shippable-qr, windows10-64-shippable, windows10-64-shippable-qr, windows7-32-shippable) regression on push 455228c96302774d192b40be9b16364affb5345f (Mon August 19 2019)

Status: RESOLVED INVALID

(Core :: CSS Parsing and Computation, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=455228c96302774d192b40be9b16364affb5345f

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

25% perf_reftest_singletons windows7-32-shippable opt e10s stylo 47.99 -> 60.15
22% perf_reftest_singletons windows10-64-shippable opt e10s stylo 47.86 -> 58.58
22% perf_reftest_singletons windows10-64-shippable-qr opt e10s stylo 48.40 -> 59.14
22% perf_reftest_singletons linux64-shippable opt e10s stylo 46.04 -> 56.20
21% perf_reftest_singletons linux64-shippable-qr opt e10s stylo 48.36 -> 58.72

Improvements:

72% perf_reftest_singletons link-style-cache-1.html windows10-64-shippable opt e10s stylo 2,712.89 -> 773.00
71% perf_reftest_singletons link-style-cache-1.html windows10-64-shippable-qr opt e10s stylo 2,685.51 -> 774.97
68% perf_reftest_singletons inline-style-cache-1.html windows10-64-shippable-qr opt e10s stylo 5,442.41 -> 1,716.84
68% perf_reftest_singletons inline-style-cache-1.html windows10-64-shippable opt e10s stylo 5,429.28 -> 1,727.26
67% perf_reftest_singletons inline-style-cache-1.html linux64-shippable opt e10s stylo 4,355.10 -> 1,427.53
67% perf_reftest_singletons inline-style-cache-1.html linux64-shippable-qr opt e10s stylo 4,489.91 -> 1,487.58
66% perf_reftest_singletons link-style-cache-1.html linux64-shippable opt e10s stylo 2,065.25 -> 711.32
65% perf_reftest_singletons link-style-cache-1.html linux64-shippable-qr opt e10s stylo 2,115.87 -> 742.84

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=22530

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/TestEngineering/Performance/Talos

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Emilio Cobos Álvarez [:emilio]]

This is adding Talos tests, so I think this is expected unless I'm missing something? I thought we only tracked the individual tests nowadays?

Anyhow I think this is INVALID.

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Yeah, looking on the other platforms reveals that some other revision caused this. Thanks!

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)

This is adding Talos tests, so I think this is expected unless I'm missing something? I thought we only tracked the individual tests nowadays?

Anyhow I think this is INVALID.

(In reply to Alexandru Ionescu :alexandrui from comment #2)

Yeah, looking on the other platforms reveals that some other revision caused this. Thanks!

Actually that's not right. If you look at this graph, revision 455228c96302 was caused by the same bug as ff3ba7a16c21 (they have only one bug in common and the evolution of the graph is similar) which we know was caused by 1480146.
I am not saying this is not expected, I'm saying that this is not INVALID. I found 2 other alerts showing what this shows. So unless you have another explaination, I'm changing the status to WON'T FIX.

Comment 4

[Emilio Cobos Álvarez [:emilio]]

So the history of that bug goes like this:

• Bug lands, with tests that are very expensive and makes some machines timeout.
• Bug gets backed out for those timeouts. This brings the total number of perf_reftest_singletons to regular levels, but since the other tests disappear (per backout) they remain with the old measurements.
• Bug relands with tests significantly simplified to accommodate for the machines that couldn't run them. perf_reftest_singletons regresses, but the tests improve because of they're both doing less work.
• ... which explains the alert in comment 0.
• ... which means that there's no actionable code change to be made, and that this regression is not a real perf regression, it's just a set of tests taking more time because there are more tests.

As long as we agree on that, I don't care if the bug is resolved as INVALID or WONTFIX :)