1631757 - 9.71 - 29.08% tart / tscrollx (windows10-64-shippable-qr) regression on push 711080298194a98751738a1db9f6d87252ad081b (Mon April 20 2020)

Status: RESOLVED INVALID

(Core :: DOM: Navigation, defect, P3)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=711080298194a98751738a1db9f6d87252ad081b

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

29% tscrollx windows10-64-shippable-qr opt e10s stylo 0.83 -> 1.07
10% tart windows10-64-shippable-qr opt e10s stylo 2.54 -> 2.79

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=25653

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/TestEngineering/Performance/Talos

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Paul Bone [:pbone]]

Looking at the graph for tscrollx, the test before mine was unusually fast, my push returned the tscrollx statistic back to the mean.

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=e3fa2adaae1f&highlightedRevisions=711080298194&series=autoland,1927528,1,1&timerange=1209600&zoom=1587291012544,1587445716373,1.4833040068039574,6.170469123084799

There was a similar change in the tart graph, but there is still an increase from the earlier pushes to mine and the later ones.

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=0&highlightedRevisions=e3fa2adaae1f&highlightedRevisions=711080298194&series=autoland,1927502,1,1&timerange=1209600&zoom=1587167732931,1587396162803,0,7.699739061848074

Comment 2

[Paul Bone [:pbone]]

I can't reproduce the regression (I wanted to reproduce it and profile it to understand it) but the instructions at https://wiki.mozilla.org/TestEngineering/Performance/Talos/Tests#tart don't work. That link to the addon doesn't work, and when I build an .xpi file and install it then the chrome://tart/content/tart.html link doesn't work.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

Could you at least verify that the patch caused the regression by pushing to try server with and without the patch?
#perf channel on matrix could help with running the tests.

(tart and tscroll are rather weird tests since they use asap mode, which isn't anything exposed to the web)

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Paul Bone [:pbone] from comment #1)

Looking at the graph for tscrollx, the test before mine was unusually fast, my push returned the tscrollx statistic back to the mean.

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&highlightedRevisions=e3fa2adaae1f&highlightedRevisions=711080298194&series=autoland,1927528,1,1&timerange=1209600&zoom=1587291012544,1587445716373,1.4833040068039574,6.170469123084799

There was a similar change in the tart graph, but there is still an increase from the earlier pushes to mine and the later ones.

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=0&highlightedRevisions=e3fa2adaae1f&highlightedRevisions=711080298194&series=autoland,1927502,1,1&timerange=1209600&zoom=1587167732931,1587396162803,0,7.699739061848074

Sorry for the late reply. I did some retriggers and I'm going to back-out and push to try to see if your patch overlapped with some infra changes that caused this.

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

The retriggers revealed that Bug 1631312 is the culprit. Thank you for rising this.

Comment 6

[Paul Bone [:pbone]]

So far what I know is that this is a real regression, at least for tart:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=851e8422329a038c55aeec0b970be8b47949d315&newProject=try&newRevision=9cbefb0f4991038b8da13d3f425456b4ad0488ca&framework=1

That compares central-with-backout and central.

I haven't been able to reproduce locally though (Linux). the initial reported regression was on Windows and that's what I tried above (I have now triggered some Linux jobs).

Comment 7

[Paul Bone [:pbone]]

Oh weird. I still see a regression from my patch for tart. maybe the two regressions reported here are unrelated.

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

The comparison indeed looks like a regression, but the gap is filled with datapoints, which makes the test just unstable.

Comment 9

[Paul Bone [:pbone]]

Okay thanks Alexandru.