1381533 - 5.07 - 11.07% tcanvasmark / tresize / tsvgr_opacity / tsvgx (windows7-32) regression on push a16d92beb70daa33c4e29bf25300ca259ed61b2a (Tue Jul 11 2017)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=a16d92beb70daa33c4e29bf25300ca259ed61b2a

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 11%  tcanvasmark summary windows7-32 pgo e10s     9,537.92 -> 8,481.67
  8%  tresize windows7-32 pgo e10s                 10.83 -> 11.68
  7%  tsvgx summary windows7-32 pgo e10s           436.51 -> 467.25
  5%  tsvgr_opacity summary windows7-32 pgo e10s   349.54 -> 367.25


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=7958

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

these tests became very noisy, the regression is the median value of the increased noise.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

looking in detail, I am not able to see any regression in opt, only pgo.  Likewise no difference in win10.

2 of the 4 regressions show a clear change on:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=c1e41e79846500d4573052b0e3e64375c8cc7434&tochange=31450261d6f7e7c7a1e67174d51947da3094aeb3

and the other 2 show it 2 pushes later:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=fdcb00d8e0c4ab49f8d3939a8547eebae5fe6d35&tochange=dec7cb09336ee273f362ce0550a36ef70d5202d3

Given the 20 retriggers we have for many of the pushes in the general range of this, I am going to suggest that we are dealing with odd pgo fallout, but that is a theory.

:ted, do you have thoughts here on how pgo or build state could be affecting the perf results to become so noisy on pgo only as of Monday July 10th (1 week ago) ?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the noise is getting "better" around July 19th- magic numbers

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I see these improvements so far:
== Change summary for alert #8096 (as of July 19 2017 08:04 UTC) ==

Improvements:

 16%  tcanvasmark summary windows7-32 pgo e10s     8,338.29 -> 9,688.46
  8%  tsvgx summary windows7-32 pgo e10s           470.36 -> 430.86
  7%  tresize windows7-32 pgo e10s                 12.07 -> 11.18

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=8096

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

possibly we can resolve this as fixed?  :igoldan, any thoughts on that?

Comment 6

[Ionuț Goldan [:igoldan]]

Yes, I agree with closing this as RESOLVED.

But I would file another bug for tresize, because there's a visible regression between July 18 and 19th:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bautoland,a14119c4d02daaf55113e3945c4385f4c927da27,1,1%5D&series=%5Bautoland,4ac681a39a4caefb56468c5bc86fa23b8cee4c4f,1,1%5D
It happens on both opt and pgo.

On the 1st look, it seems it's just a typical, separate regression than this one.
I'll do some retriggers and backfills, to properly identify it.