1323057 - 2.13 - 13.33% tabpaint / tart / tps / tresize / tsvg_static (windows7-32, windows8-64) regression on push 6fddc51e9d1530484e7875ea6b67141497db784d (Thu Dec 8 2016)

Status: RESOLVED WONTFIX

(Firefox :: Tabbed Browser, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push 6fddc51e9d1530484e7875ea6b67141497db784d. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 13%  tresize windows8-64 opt                 11.69 -> 13.25
  8%  tart summary windows8-64 pgo            5.29 -> 5.72
  5%  tart summary windows8-64 opt            6.63 -> 6.98
  5%  tsvg_static summary windows8-64 opt     68.14 -> 71.39
  3%  tart summary windows7-32 opt            7.35 -> 7.6
  3%  tart summary windows8-64 pgo e10s       5.73 -> 5.91
  3%  tps summary windows7-32 opt             72.47 -> 74.53
  3%  tps summary windows8-64 opt             67.34 -> 69.23
  3%  tabpaint summary windows7-32 opt        89.18 -> 91.66
  3%  tart summary windows7-32 opt e10s       7.95 -> 8.17
  3%  tart summary windows8-64 opt e10s       7.06 -> 7.25
  3%  tart summary windows7-32 pgo e10s       5.93 -> 6.09
  2%  tps summary windows7-32 pgo             55.23 -> 56.41


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=4474

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

here is a perfherder comparison I did after many retriggers:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=a3945af43157&newProject=mozilla-inbound&newRevision=6fddc51e9d15&framework=1

overall, these match up well for the tests/platforms that we generated alerts for, there are some differences in severity.

:dao, I see you are the patch author, some of these are probably expected, others might not be.  Could you comment on what your expectations are for bug 658467 and what options you think are reasonable for fixing or accepting these regressions?

Comment 2

[Dão Gottwald [:dao]]

So most regressions are for non-e10s. That's good, it shows that e10s saves us from fancy UI effects affecting web content. I think we'll need to have a discussion about how much we care about non-e10s talos regressions going forward as we switch more and more users to e10s.

All e10s regressions are limited to tart and pretty small. As I said in bug 658467, I think we can accept those as the cost of fading out tab labels.

Comment 3

[Dão Gottwald [:dao]]

(In reply to Dão Gottwald [:dao] from comment #2)
> So most regressions are for non-e10s.

And by far the worst is tresize. This makes sense, as resizing the window resizes tabs too, but I don't think this matters much in terms of when and how users will perceive slower performance as a result of bug 658467.

Overall this looks reasonable. I don't think we can do anything about these regressions other than backing out bug 658467, which IMHO we shouldn't.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for the great information here.  We can mark this as wontfix and move forward.  You brought an interesting point up regarding caring about non-e10s talos results, that is a topic we should pursue and consider making decisions about this soon.