1272124 - 2.25 - 7.73% cart / tp5o % Processor Time / tp5o_scroll e10s (windows7-32, windows8-64) regression on push b1c6143527c0 (Tue May 10 2016)

Status: RESOLVED WONTFIX

(Firefox :: Untriaged, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push b1c6143527c0:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=c31a5ad159946c74a32c9ebedcaa8585593ff178&tochange=b1c6143527c0edd015cf9d890a2cb0d61c9aa053

As author of one of the patches included in that push, we need your help to address this regression.

This is a list of all known regressions and improvements related to the push:
https://treeherder.mozilla.org/perf.html#/alerts?id=1142

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see:
https://wiki.mozilla.org/Buildbot/Talos/Tests#tp5
https://wiki.mozilla.org/Buildbot/Talos/Tests#tp5o_scroll
https://wiki.mozilla.org/Buildbot/Talos/Tests#TART.2FCART

Reproducing and debugging the regression:

If you would like to re-run this Talos test on a potential fix, use try with the following syntax:
try: -b o -p win32,win64 -u none -t tp5o[Windows 7,Windows 8],g1-e10s[Windows 7,Windows 8],svgr-e10s[Windows 7,Windows 8] --rebuild 5  # add "mozharness: --spsProfile" to generate profile data

(we suggest --rebuild 5 to be more confident in the results)

To run the test locally and do a more in-depth investigation, first set up a local Talos environment:
https://wiki.mozilla.lorg/Buildbot/Talos/Running#Running_locally_-_Source_Code

Then run the following command from the directory where you set up Talos:
talos --develop -e [path]/firefox -a tp5o:tp5o_scroll:cart --e10s

Making a decision:
As the patch author we need your feedback to help us handle this regression.
*** Please let us know your plans by Monday, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations:
https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I did a lot of retriggers to find the root cause here- and a compare view to show the differences:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=c31a5ad15994&newProject=mozilla-inbound&newRevision=b1c6143527c0&framework=1&filter=tp5

:mattwoodrow, can you look at this and help determine if this is expected or what we might be able to do in order to reduce this regresssion?

Comment 2

[Matt Woodrow (:mattwoodrow)]

It's almost certainly this changeset: https://hg.mozilla.org/integration/mozilla-inbound/rev/666c10f1dd52

This makes us block the compositor until the previous frames work has been completed by the GPU.

This reduces the number of frames queued in the GPU pipeline, so should reduce latency to the screen and more accurately reflect what we actually want to measure.

I think given that these regressions are acceptable.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks Matt!  I am fine marking this as resolved/wontfix.  Sometimes :avih has a few opinions- lets see if he has anything to add.

Comment 4

[Avi Halachmi (:avih)]

Trying to figure out the exact regressions we're supposedly/reportedly having, because there's only that much info I can take from the title.

Looking at the alerts view at comment 0, and ignoring the striked-through lines (should I ignore them? if yes, it would be nice if there was an option to hide them), I can see the following regressions (and no improvements):

 3% tp5o process time   win7-32 opt
20% tp5o responsiveness win7-32 opt
10% tp5o summary        win7-32 opt (and e10s too)
 7% tp5o_scroll         win8-64 (opt and pgo)

And I'm not seeing CART regressions there.

Looking at the compare view from comment 1 (with reasonable amount of retriggers), I see only two lines marked as meaningful:

2% tp5o processor time win7-32 opt e10s
7% tp5o_scroll         win8-64 opt e10s

So what regressions are we reporting here exactly?

Comment 5

[Avi Halachmi (:avih)]

Also, the title currently is:
> 2.25 - 7.73% cart / tp5o % Processor Time / tp5o_scroll e10s (windows7-32, windows8-64)

But CART does not appear anywhere, and processor time appears the least of the regressions however you look at it and it appears right after CART, while responsiveness, which is supposedly the highest regression according to the alert doesn't even appear at the title...

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

the regressions are listed nicely in the compare view, cart showed up because retriggers showed this having a difference:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&zoom=1462902959640.0862,1462934012459.0518,36.86431226765799,39.07620817843866

It looks like tp5o % processor time (win7) and tp5o_scroll (win8) are the main two which are related to this change.

Comment 7

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #6)
> It looks like tp5o % processor time (win7) and tp5o_scroll (win8) are the
> main two which are related to this change.

Assuming the numbers I posted at comment 4 are correct, the processor time regression is 2% which is relatively small.

Which leaves us with 7% tp5o_scroll on windows 8-64 e10s opt (and possibly pgo - no numbers there) only, as far as I can tell. Do correct me if I'm wrong.

This, together with the fact that the change fixes racy behavior with some videos, makes me think we should accept it, unless Matt think it can still be improved.

Matt, please WONTFIX if you don't have plans to improve it. Thanks.

Comment 8

[Avi Halachmi (:avih)]

> This, together with the fact that the change fixes racy behavior with some videos,

And also, after discussion with Matt, it seems the change mostly limits the composition rate, which while can affect the scroll test and is not meaningless, is less of a bottleneck in most real world scenarios.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

these changes are uplifted to beta:
https://treeherder.mozilla.org/perf.html#/alerts?id=1371

and aurora:
https://treeherder.mozilla.org/perf.html#/alerts?id=1357

These are processor time and tart regressions, I suspect these fall under the same wontfix classification as we did for trunk.