1246399 - 3-12% Linux 64/Win*/MacOS* tsvgr_opacity/tsvgx/sessionrestore/sessionrestore_no_auto_restore/tp5o/ts_paint regression on Fx-Team (v.47) on Feb 04, 2016 from push f5bd25c789b6

Status: RESOLVED FIXED

(DevTools :: JSON Viewer, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from your commit f5bd25c789b6bab468de449782b4985e39239968 in bug 1245121.  We need you to address this regression.

This is a list of all known regressions and improvements related to your bug:
http://alertmanager.allizom.org:8080/alerts.html?rev=f5bd25c789b6bab468de449782b4985e39239968&showAll=1

On the page above you can see Talos alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test, please see: https://wiki.mozilla.org/Buildbot/Talos/Tests#tsvg-opacity

Reproducing and debugging the regression:
If you would like to re-run this Talos test on a potential fix, use try with the following syntax:
try: -b o -p linux64,win64,win32,macosx64 -u none -t svgr  # add "mozharness: --spsProfile" to generate profile data

To run the test locally and do a more in-depth investigation, first set up a local Talos environment:
https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code

Then run the following command from the directory where you set up Talos:
talos --develop -e <path>/firefox -a tsvgr_opacity

Making a decision:
As the patch author we need your feedback to help us handle this regression.
*** Please let us know your plans by Tuesday, or the offending patch will be backed out! ***

Our wiki page outlines the common responses and expectations:
https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Here is a compare view of this change vs the previous one:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=fx-team&originalRevision=46d6d868ba0f&newProject=fx-team&newRevision=f5bd25c789b6&framework=1

it is a lot of regressions- :jryans, can you take this bug and find a fix or determine what we would lose if we backout?

Comment 2

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Honza, is this something you can investigate?  We may need to revise how the content is intercepted when loading...  it seems to be affecting non-JSON loads quite a lot from these numbers.

Joel, we've been shipping this feature preffed on for Dev. Edition only for a bit now, originally in bug 1132203.  (There was a small Talos regression for fx-team detected at the time in bug 1214629, but the feature would actually have been *disabled* in such a build.)  So, I am guessing we likely shipped these same perf hits to Dev. Edition 44 and later, but did not notice it perhaps?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I don't see much evidence of this showing up on aurora and not on central:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=31536000&series=[mozilla-aurora,232ba0c786c563852749be61de906cd21c1dc7a5,1]&series=[mozilla-central,232ba0c786c563852749be61de906cd21c1dc7a5,1]&highlightedRevisions=46d6d868ba0f&highlightedRevisions=f5bd25c789b6

but that does sound plausible that this has existed on aurora and now we are seeing it on nightly.

Honza, please let me know how I can help.

Comment 4

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I don't know what could be wrong, but here is a patch changing the way how JS modules are loaded, let's see if it helps.

Try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9f24802ed6cd

Talos:
https://treeherder.mozilla.org/perf.html#/comparechooser?newProject=try&newRevision=9f24802ed6cd

Honza

Comment 5

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Here is another push, now with all talos tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8d7005065842
https://treeherder.mozilla.org/perf.html#/comparechooser?newProject=try&newRevision=8d7005065842

Joel, can you take a look at the results as soon as available?
Btw. I never know how to properly use the compare-chooser. What base should I use?

Honza

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:honza,  thanks for the try push.  The compare chooser can be confusing, we have new code which finds the base automatically in staging and will be deployed later this week, but until then- it is an unfriendly process.

I have found it for you:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=fx-team&originalRevision=49945e5fe79e&newProject=try&newRevision=9f24802ed6cd&framework=1

in addition, I retriggered the jobs on the base and the try push so results should be in within a couple of hours.

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Joel's link comment 6 is based on the first try run, not the one with all Talos in comment 5 (since more than just tsvgr is impacted).

Here's a compare with the larger run:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=fx-team&originalRevision=49945e5fe79e&newProject=try&newRevision=8d7005065842&framework=1

Since this compare is against the revision just before it and shows no improvement, I believe it means this potential fix did not help.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

keep in mind we are comparing a single data point in most of these cases.  I have retriggered all the jobs for base/try and we should have something much more realistic to work with.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

any progress here?  this is a lot of regressions!

Comment 10

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I don't have any immediate solutions. We need more time for figuring out what exactly is causing the problem and we might also need to revisit the way how the content is intercepted and transforme. I am suggesting to back out the patch for now and continue looking for solutions.

Honza

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Okay, I have backed out the original change:

https://hg.mozilla.org/integration/fx-team/rev/ef2559c8d203

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

alright, please ask for help if there is any I can provide.  If understanding a talos test, how to run, how to interpret results, or how to compare results- I would love to help where I can.

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Looks like the back out has resolved the issue for now.  :jmaher, do you agree?

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

yes, i verified about half the data points on the graphs!