1401127 - 2.23 - 61.65% tp5o_webext regressions on push aca18d5ec8efca82f3e94db8f31cff578e69fa10 (Fri Sep 15 2017)

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=aca18d5ec8efca82f3e94db8f31cff578e69fa10

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 62%  ts_paint_webext linux64 pgo e10s     854.23 -> 1,380.83
 61%  tp5o_webext responsiveness linux64 pgo e10s5.41 -> 8.70
 61%  tp5o_webext responsiveness linux64 opt e10s6.03 -> 9.69
 58%  tp5o_webext summary osx-10-10 opt e10s266.16 -> 421.01
 52%  ts_paint_webext linux64 opt e10s     944.29 -> 1,439.25
 51%  tp5o_webext responsiveness windows10-64 pgo e10s3.81 -> 5.75
 49%  tp5o_webext responsiveness windows10-64 opt e10s4.29 -> 6.39
 43%  tp5o_webext summary linux64 pgo e10s 251.19 -> 360.42
 41%  tp5o_webext summary linux64 opt e10s 297.85 -> 419.42
 38%  tp5o_webext summary windows10-64 opt e10s274.96 -> 380.54
 38%  tp5o_webext summary windows7-32 pgo e10s219.18 -> 301.83
 37%  tp5o_webext summary windows10-64 pgo e10s223.99 -> 307.72
 37%  tp5o_webext responsiveness windows7-32 opt e10s4.70 -> 6.45
 36%  tp5o_webext responsiveness windows7-32 pgo e10s3.77 -> 5.13
 34%  tp5o_webext summary windows7-32 opt e10s304.69 -> 407.77
 20%  tp5o_webext % Processor Time windows7-32 opt e10s56.41 -> 67.43
 18%  tp5o_webext Private Bytes windows7-32 pgo e10s152,425,778.63 -> 180,204,989.92
 15%  tp5o_webext Private Bytes windows7-32 opt e10s154,287,366.88 -> 177,443,498.39
 14%  tp5o_webext Main_RSS linux64 pgo e10s187,598,612.19 -> 213,456,847.05
 13%  tp5o_webext Main_RSS linux64 opt e10s194,286,897.76 -> 219,431,603.76
 12%  tp5o_webext Main_RSS osx-10-10 opt e10s352,973,998.14 -> 396,425,659.79
 11%  tp5o_webext Main_RSS windows7-32 pgo e10s128,619,324.46 -> 143,204,995.82
  9%  tp5o_webext Main_RSS windows7-32 opt e10s133,500,824.08 -> 145,165,425.15
  4%  tp5o_webext Private Bytes linux64 pgo e10s1,018,622,305.81 -> 1,056,033,445.52
  2%  tp5o_webext Private Bytes linux64 opt e10s1,019,742,992.52 -> 1,042,473,129.94


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=9471

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

To my understading, these should represent the new baseline for our tp5o_webext tests. Am I right?
As you noted here, Kris:
https://bugzilla.mozilla.org/show_bug.cgi?id=1381961#c10
many tests here have come back to original baseline after the merge with bug 1381961, while the rest went down partially.

Unless you find something unexpected in the regression list above, I would resolve this as WONTFIX.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

some of the values end up being reset fully or partially with bug 1381961 - maybe that is fixing some of the issues here?

Comment 3

[Kris Maglione [:kmag]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> To my understading, these should represent the new baseline for our
> tp5o_webext tests. Am I right?

Yes, this is the new baseline. The changes represent intentional extra overhead added to the talos tests so that we can measure the actual performance impact of more of our code, rather than an actual performance regression.

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)
> some of the values end up being reset fully or partially with bug 1381961 -
> maybe that is fixing some of the issues here?

The interaction with bug 1381961 is complicated. Those changes improved baseline performance in a lot of areas considerably. So the new baseline with the more complicated talos webextension is similar to the old overhead with the trivial extension, but is still worse than the baseline without the extension.

That said, bug 1381961 also disproportionately affected a lot of WebExtension framework code, so the gains in the webext tests are more than just the baseline gains. This is why I wanted to land these talos changes before bug 1381961 landed, so we get a better picture of the impact it had.