5.07 - 11.07% tcanvasmark / tresize / tsvgr_opacity / tsvgx (windows7-32) regression on push a16d92beb70daa33c4e29bf25300ca259ed61b2a (Tue Jul 11 2017)

RESOLVED FIXED

Status

defect
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: igoldan, Unassigned)

Tracking

({perf, regression, talos-regression})

Firefox Tracking Flags

(Not tracked)

Details

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=a16d92beb70daa33c4e29bf25300ca259ed61b2a

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 11%  tcanvasmark summary windows7-32 pgo e10s     9,537.92 -> 8,481.67
  8%  tresize windows7-32 pgo e10s                 10.83 -> 11.68
  7%  tsvgx summary windows7-32 pgo e10s           436.51 -> 467.25
  5%  tsvgr_opacity summary windows7-32 pgo e10s   349.54 -> 367.25


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=7958

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling
these tests became very noisy, the regression is the median value of the increased noise.
Component: Untriaged → General
Product: Firefox → Release Engineering
looking in detail, I am not able to see any regression in opt, only pgo.  Likewise no difference in win10.

2 of the 4 regressions show a clear change on:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=c1e41e79846500d4573052b0e3e64375c8cc7434&tochange=31450261d6f7e7c7a1e67174d51947da3094aeb3

and the other 2 show it 2 pushes later:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=fdcb00d8e0c4ab49f8d3939a8547eebae5fe6d35&tochange=dec7cb09336ee273f362ce0550a36ef70d5202d3

Given the 20 retriggers we have for many of the pushes in the general range of this, I am going to suggest that we are dealing with odd pgo fallout, but that is a theory.

:ted, do you have thoughts here on how pgo or build state could be affecting the perf results to become so noisy on pgo only as of Monday July 10th (1 week ago) ?
Flags: needinfo?(ted)
the noise is getting "better" around July 19th- magic numbers
I see these improvements so far:
== Change summary for alert #8096 (as of July 19 2017 08:04 UTC) ==

Improvements:

 16%  tcanvasmark summary windows7-32 pgo e10s     8,338.29 -> 9,688.46
  8%  tsvgx summary windows7-32 pgo e10s           470.36 -> 430.86
  7%  tresize windows7-32 pgo e10s                 12.07 -> 11.18

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=8096
possibly we can resolve this as fixed?  :igoldan, any thoughts on that?
Flags: needinfo?(ionut.goldan)
Yes, I agree with closing this as RESOLVED.

But I would file another bug for tresize, because there's a visible regression between July 18 and 19th:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bautoland,a14119c4d02daaf55113e3945c4385f4c927da27,1,1%5D&series=%5Bautoland,4ac681a39a4caefb56468c5bc86fa23b8cee4c4f,1,1%5D
It happens on both opt and pgo.

On the 1st look, it seems it's just a typical, separate regression than this one.
I'll do some retriggers and backfills, to properly identify it.
Flags: needinfo?(ionut.goldan)
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Flags: needinfo?(ted)
You need to log in before you can comment on or make changes to this bug.