1340873 - 2.01 - 3.03% sessionrestore / sessionrestore_no_auto_restore / ts_paint (linux64, osx-10-10, windows8-64) regression on push 29f4d23a9e97c324b40c63c71eff04add5fd2a09 (Thu Feb 16 2017)

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push 29f4d23a9e97c324b40c63c71eff04add5fd2a09. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  3%  ts_paint windows8-64 pgo                            808.92 -> 833.42
  3%  ts_paint linux64 pgo                                1139.75 -> 1172.08
  3%  sessionrestore_no_auto_restore windows8-64 pgo      719.08 -> 739.08
  3%  ts_paint windows8-64 pgo e10s                       802.17 -> 824
  3%  ts_paint osx-10-10 opt                              1191.25 -> 1222.75
  3%  sessionrestore windows8-64 pgo                      696.04 -> 713.83
  2%  ts_paint windows8-64 opt e10s                       951.08 -> 974
  2%  ts_paint linux64 opt                                1333.33 -> 1364.42
  2%  ts_paint linux64 opt e10s                           1237.75 -> 1266.33
  2%  ts_paint windows8-64 opt                            960.5 -> 981.58
  2%  sessionrestore_no_auto_restore windows8-64 opt      862.42 -> 880.42
  2%  sessionrestore windows8-64 opt                      839.04 -> 856
  2%  ts_paint linux64 pgo e10s                           1074.88 -> 1096.5

Improvements:

 12%  tp5n main_startup_fileio windows7-32 pgo          70278447.25 -> 61793715.08
 12%  tp5n main_startup_fileio windows7-32 pgo e10s     70382523.58 -> 61987080.42
  5%  tp5o Main_RSS linux64 pgo e10s                    170999472.55 -> 163160602.48
  4%  tp5o Main_RSS linux64 opt e10s                    173635785.43 -> 166856555
  3%  tp5o Main_RSS linux64 pgo                         249584310.97 -> 241745580.81
  3%  tp5o Main_RSS linux64 opt                         245844351.87 -> 238333579.64
  2%  tp5o Main_RSS osx-10-10 opt                       397510251.14 -> 388817320.26


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=5164

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

here is a comparison view of this commit and the previous one:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=autoland&originalRevision=e76a72dceeea&newProject=autoland&newRevision=29f4d23a9e97c324b40c63c71eff04add5fd2a09&framework=1

it shows the same regressions as the alerts found.

:catlee, as the patch author here, can you figure out why this is happening and what we should do to resolve this?  Either fix it completely, fix it partially, back out the patch, or accept the regression.

Comment 2

[Chris AtLee [:catlee]]

This is expected due to backing out the omni.ja decompression patch in bug 1340157.

We'd like to measure the effect again on Talos and Telemetry before making a final decision of whether to enable it on all branches, or keep it disabled.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

what is the timeline for measuring this and marking this as wontfix or reverting these changes?

Comment 4

[Chris AtLee [:catlee]]

I'd like to make a decision by the end of this week. Telemetry data is still coming in.