1290808 - 0.7 - 2.44% cart (windows7-32, windows8-64) regression on push c3565c8b1cdb575db1c80c7791984a6490598b84 (Sat Jul 30 2016)

Status: RESOLVED WONTFIX

(Hello (Loop) :: Client, defect)

Description

[Alison Shiue]

Talos has detected a Firefox performance regression from push c3565c8b1cdb575db1c80c7791984a6490598b84. As author of one of the patches included in that push, we need your help to address this regression.

Summary of tests that regressed:

  cart summary windows7-32 opt e10s - 2.44% worse
  cart summary windows8-64 opt e10s - 0.7% worse

Summary of tests that improved:

  ts_paint linux64 opt - 2.06% better
  ts_paint linux64 opt e10s - 2.32% better
  ts_paint windows7-32 opt e10s - 2.03% better
  ts_paint windows8-64 opt e10s - 3.6% better


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=2166

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Alison Shiue]

I did some retriggers for this alert, here is the zooming better view:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bfx-team,26841cb06ef026bbd324cb82b42bb716fa5dca25,1,1%5D&series=%5Bfx-team,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&zoom=1469826994610.4543,1469902120814.3252,29.30910296983182,40

This push did a perf improvement on ts_paint tests, however, it also caused perf regression issues on some cart tests.

Hi Mark, as you are the patch author, could you take a look at this and determine what is the root cause? Thanks!

Comment 2

[Mark Banner (:standard8)]

All we've done here is to remove the Loop code - this removes a button that the Loop system add-on was previously adding to the toolbar by default.

I can't think of any reason that this would slow the cart tests down - there's now one less button to animate!

Comment 3

[Mark Banner (:standard8)]

Joel/Matt, any ideas here? I don't think this is anything Hello can own, as all we've done is remove our code.

Comment 4

[Avi Halachmi (:avih)]

This has one barely meaningful regression of 2.4% (cart win7 32 e10s), one negligible regression, and a bunch of barely meaningful improvements.

We should probably accept it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

according to the subtests:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=fx-team&originalRevision=e546cb06c2872bf90fa42d00fc311c580bd33a1f&newProject=fx-team&newRevision=c3565c8b1cdb575db1c80c7791984a6490598b84&originalSignature=bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45&newSignature=bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45&framework=1

it looks like we see a small win on enter, but a large regression on exit:
cart 2-customize-exit.half.TART opt e10s 10.32 ± 0.42% 	< 	13.94 ± 11.60% 	35.04% 

Another thought here is we did a clobber- maybe this cart regression is really from something else a few pushes earlier.

If the 2-customize-exit.half.TART doesn't help us figure it out, I am fine accepting this.

Comment 6

[Matthew N. [:MattN]]

Joel and Avi can handle this at least until we are more confident in the exact push.

Comment 7

[Mark Banner (:standard8)]

Just to note the ts_paint improvement seems to be a recovery from when Hello was initially landed as a system add-on - see bug 1230280. Unfortunately we never worked out why it regressed in the first place.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Mark,

Do we mark this as wontfix?

Comment 9

[Mark Banner (:standard8)]

(In reply to Joel Maher ( :jmaher) from comment #8)
> Do we mark this as wontfix?

I think unless we have someone that can re-identify the exact push, and then dig into the customisation code, I don't think this is going to go much further.