1414321 - 4.64 - 5.18% speedometer (linux64) regression on push e9349ad2f1f8fec862b1d2271d0d8f25ad0814d4 (Thu Nov 2 2017)

Status: RESOLVED FIXED

(Core :: Graphics, defect, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=e9349ad2f1f8fec862b1d2271d0d8f25ad0814d4

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  5%  speedometer summary linux64 pgo e10s     40.31 -> 38.22
  5%  speedometer summary linux64 opt e10s     34.28 -> 32.69


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=10328

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this has a regression on speedometer- the benchmark that we used to compare firefox vs chrome, specifically on windows10, this regression is on linux- :rhunt, can you take a look at this?

Comment 2

[Ryan Hunt [:rhunt] (on leave until early May)]

The patches were backed out from central here [1] because of a test timeout.

I'll still keep this in mind before trying to land again.

BTW, I don't see a description for speedometer under [2], what are the units? What does 40.31 -> 38.22 mean?

[1] https://hg.mozilla.org/mozilla-central/rev/28647c01f828
[2] https://wiki.mozilla.org/Buildbot/Talos/Tests

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

here is the main website for speedometer:
http://browserbench.org/Speedometer/

and on github:
https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer

I filed bug 1414331 to track describing the test, we will get that fixed soon.

thanks for the heads up on the backout- I am happy to close this bug out, or leave it open if you wish to keep track of the regression until this lands again.

Comment 4

[Ryan Hunt [:rhunt] (on leave until early May)]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3)
> here is the main website for speedometer:
> http://browserbench.org/Speedometer/
> 
> and on github:
> https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer
> 
> I filed bug 1414331 to track describing the test, we will get that fixed
> soon.
> 
> thanks for the heads up on the backout- I am happy to close this bug out, or
> leave it open if you wish to keep track of the regression until this lands
> again.

Let's leave this open to keep track of the regression before I try and land it again. Thanks!

Comment 5

[Ben Kelly [:bkelly, not reviewing]]

Is this really for speedometer v1 and not v2 like we use on awfy?  Comment 3 does not point to the same versions we use on awfy.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I believe we have the latest version in talos, we use what is in-tree:
http://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/Speedometer

Comment 7

[Ionuț Goldan [:igoldan]]

Be aware that new regressions have showed up for this bug:

== Change summary for alert #10328 (as of Thu, 02 Nov 2017 19:30:42 GMT) ==

Regressions:

 85%  tsvgx summary linux64 pgo e10s     355.97 -> 660.14
 81%  tsvgx summary linux64 opt e10s     376.70 -> 681.25
 31%  tscrollx summary windows10-64 opt e10s2.42 -> 3.17
 31%  tscrollx summary windows10-64 pgo e10s2.39 -> 3.14
 18%  tsvgx summary windows10-64 opt e10s237.67 -> 279.47
 17%  tsvgx summary windows10-64 pgo e10s220.68 -> 259.12
  9%  tscrollx summary linux64 opt e10s  6.57 -> 7.18
  8%  tscrollx summary linux64 pgo e10s  6.35 -> 6.88
  7%  tart summary linux64 pgo e10s      4.62 -> 4.96
  6%  tart summary linux64 opt e10s      5.17 -> 5.49
  2%  kraken summary linux64 pgo e10s    1,376.70 -> 1,407.88
  2%  kraken summary linux64 opt e10s    1,418.48 -> 1,450.34

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=10328

Comment 8

[Ionuț Goldan [:igoldan]]

Of course, these regressions were canceled by the backout. Comment 7 is just FYI.

Comment 9

[Mike Taylor [:miketaylr]]

(setting 58 to unaffected since this was backed out)

Comment 10

[Ryan Hunt [:rhunt] (on leave until early May)]

Bug 1399692 has landed in central again. I'm guessing if there are any talos regressions, they'll be opened in a new bug, so for now I'll close this.