Closed Bug 1307336 Opened 8 years ago Closed 8 years ago

2.86% cart (windows7-32) regression on push 16219621894e4ac4d222b4fdcdf04f84eff59b17 (Thu Sep 29 2016)

Categories

(Firefox :: Untriaged, defect)

52 Branch
defect
Not set
normal

Tracking

()

RESOLVED WONTFIX
Tracking Status
firefox49 --- unaffected
firefox50 --- unaffected
firefox51 --- unaffected
firefox52 --- affected

People

(Reporter: ashiue, Unassigned)

References

Details

(Keywords: perf, regression, talos-regression)

Talos has detected a Firefox performance regression from push 16219621894e4ac4d222b4fdcdf04f84eff59b17. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  3%  cart summary windows7-32 opt e10s     38.47 -> 39.56


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=3515

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling
I also did some retriggers on other platforms, but only windows7-32 shows obvious regression problem.

Here is the better zooming view:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&zoom=1475061006443.5261,1475174004158.8613,36.98507462686567,40.80597014925373&selected=%5Bmozilla-inbound,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,37177,36892148,1%5D

Hi Matt, as you are the patch author, can you take a look at this and determine what is the root cause? Thanks!
Blocks: 1302124, 1305213
Flags: needinfo?(matt.woodrow)
This was a really simple patch that just fixes a configuration option controlling hardware accelerated video decoding.

I don't see how it could possibly affect CART.
Flags: needinfo?(matt.woodrow)
looking at this in more detail, it does look as though the change did cause a tart regression:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bfx-team,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&series=%5Bmozilla-inbound,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&series=%5Bautoland,bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D

here are the subtests (a lot of change across the board):
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-inbound&originalRevision=a357f5e03549f6a8ccb7e40d8a9237661b00a0d8&newProject=mozilla-inbound&newRevision=16219621894e4ac4d222b4fdcdf04f84eff59b17&originalSignature=bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45&newSignature=bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45&framework=1

what I notice most is that we seem to have a noisier test (not quite bi-modal territory).  Is it possible that we have changed some timing somewhere, maybe some initialization in the browser?  Looking at the patch it seems as though we moved UpdateCanUseHardwareVideoDecoding() after some initialization code- possibly this is seen during runtime more than we thought?

Keep in mind our windows7 machines are older and the graphics story is not our ideal end user story- maybe this is a factor of our environment also.
(In reply to Joel Maher ( :jmaher) from comment #3)
> looking at this in more detail, it does look as though the change did cause
> a tart regression:
> https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bfx-team,
> bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&series=%5Bmozilla-inbound,
> bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D&series=%5Bautoland,
> bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45,1,1%5D
> 
> here are the subtests (a lot of change across the board):
> https://treeherder.mozilla.org/perf.html#/
> comparesubtest?originalProject=mozilla-
> inbound&originalRevision=a357f5e03549f6a8ccb7e40d8a9237661b00a0d8&newProject=
> mozilla-
> inbound&newRevision=16219621894e4ac4d222b4fdcdf04f84eff59b17&originalSignatur
> e=bf7cba00e2c43267bbfdca11eb8f1e7e50fb9f45&newSignature=bf7cba00e2c43267bbfdc
> a11eb8f1e7e50fb9f45&framework=1
> 
> what I notice most is that we seem to have a noisier test (not quite
> bi-modal territory).  Is it possible that we have changed some timing
> somewhere, maybe some initialization in the browser?  Looking at the patch
> it seems as though we moved UpdateCanUseHardwareVideoDecoding() after some
> initialization code- possibly this is seen during runtime more than we
> thought?

This code only runs during process initialization, the time spent in it shouldn't be within the recording at all.

> 
> Keep in mind our windows7 machines are older and the graphics story is not
> our ideal end user story- maybe this is a factor of our environment also.

The only runtime cost change should be that we allow hardware accelerated video decoding, but this was just fixing a regression from a few days earlier.
possibly we have to make this a wontfix?
I guess that's probably the best idea, it's not a huge regression.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.