1259919 - 2% tcanvasmark pgo (windows7-32, windows8-64, windowsxp) regression on push 8d59e338a5bd (Thu Mar 24 2016)

Status: RESOLVED DUPLICATE of bug 1259923

(Core :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push 8d59e338a5bd. As author of one of the patches included in that push, we need your help to address this regression.

This is a list of all known regressions and improvements related to the push:
https://treeherder.mozilla.org/perf.html#/alerts?id=612

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see:
https://wiki.mozilla.org/Buildbot/Talos/Tests#CanvasMark

Reproducing and debugging the regression:

If you would like to re-run this Talos test on a potential fix, use try with the following syntax:
try: -b o -p win32,win64 -u none -t chromez[Windows XP,Windows 7,Windows 8],chromez-e10s[Windows XP,Windows 7,Windows 8] --rebuild 5  # add "mozharness: --spsProfile" to generate profile data

(we suggest --rebuild 5 to be more confident in the results)

To run the test locally and do a more in-depth investigation, first set up a local Talos environment:
https://wiki.mozilla.lorg/Buildbot/Talos/Running#Running_locally_-_Source_Code

Then run the following command from the directory where you set up Talos:
talos --develop -e [path]/firefox -a tcanvasmark

(add --e10s to run tests in e10s mode)

Making a decision:
As the patch author we need your feedback to help us handle this regression.

Our wiki page outlines the common responses and expectations:
https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:gps, I thought this was related to the vs2015 compiler change, but it looks to be on the push prior.  In fact all the performance changes are on pgo for vs2015 except for tcanvasmark opt which has slight wins.  Maybe this is something to do with a clobber or something else?

It would be nice to figure this out.

Comment 2

[Gregory Szorc [:gps]]

I don't see how this could possibly be related to bug 1256027. Since it is a PGO regression, my guess is it is a regression between this commit (https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=8d59e338a5bd) and the last PGO builds on inbound from before. That would be https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b74d0709e0a5. But it looks like you triggered extra Talos jobs on that and I don't see how the commits in between could possibly cause a performance change. WTF.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I have done some try pushes, before/after:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=2d29268052e7&tochange=8613b844e390

lets see if this reproduces or not.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

here is the compare:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=8613b844e390&newProject=try&newRevision=2d29268052e7&framework=1&showOnlyImportant=0

the regression isn't showing, this must be related to the actual vs2015 change- odd that it showed up in the push prior.  In this case, maybe this is one of those issues with per build differences.

:wlach, as I will be on pto next week, can you handle sorting out the alerts in perfherder so we have this catalogued properly- my recommendation is to duplicate this bug to bug 1259923.

Comment 5

[William Lachance (:wlach)]

Yeah, let's mark this one as a duplicate.