6.16% tart (windows7-32) regression on push ea326f6e7e6c (Sat Sep 17 2016)

RESOLVED DUPLICATE of bug 1302727

Status

()

Firefox
Untriaged
RESOLVED DUPLICATE of bug 1302727
a year ago
a year ago

People

(Reporter: jmaher, Assigned: billm)

Tracking

({perf, regression, talos-regression})

51 Branch
perf, regression, talos-regression
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox49 unaffected, firefox50 unaffected, firefox51 fixed, firefox52 fixed)

Details

(Reporter)

Description

a year ago
Talos has detected a Firefox performance regression from push eaf5eb6f8fa. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  6%  tart summary windows7-32 opt e10s     7.49 -> 7.95


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=3295

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling
(Reporter)

Comment 1

a year ago
:billm, I see you are the patch author here can you take a look at this regression and help find the root cause.  This was landed right before we uplifted on Monday (and right after another regression, so we had missed it)
Flags: needinfo?(wmccloskey)
(Reporter)

Updated

a year ago
Summary: 6.16% tart (windows7-32) regression on push eaf5eb6f8fa (Sat Sep 17 2016) → 6.16% tart (windows7-32) regression on push ea326f6e7e6c (Sat Sep 17 2016)
(Reporter)

Comment 2

a year ago
and I filed the bug on the downstream alert, here is the proper alert we should worry about for tracking in perfherder:
== Change summary for alert #3322 (as of September 21 2016 03:19 UTC) ==

Regressions:

 10%  tart summary windows7-32 opt e10s     7.11 -> 7.86

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=3322
Assignee: nobody → wmccloskey
Flags: needinfo?(wmccloskey)
status-firefox49: --- → unaffected
status-firefox50: --- → unaffected
status-firefox51: --- → affected
status-firefox52: --- → affected
(Reporter)

Comment 3

a year ago
:billm, do you have any questions about this regression?  Any luck coming up with a fix?
Flags: needinfo?(wmccloskey)
Hi Joel,
I'm having some trouble understanding the magnitude of the regression here. Bug 1302727 landed shortly before my patch and it also regressed TART. I think the right comparison to make is this one:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=c6876f222943&newProject=mozilla-inbound&newRevision=ea326f6e7e6c485907fdb4e1739bcb2707f7c1bc&framework=1&showOnlyImportant=0

since revision c6876f222943 landed after bug 1302727. I do see a 3.90% regression, but the confidence is "low".

I also looked at the graph on inbound:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,5c04c558cb10f77bcf9a38249e74b2705d6b51b9,1,1%5D&zoom=1474000846534.81,1474155300541.1392,6.241635687732343,8.20446096654275

It seems like bug 1302727 took us from ~6.9 to ~7.5. Then my patch took us to ~7.9. Can you confirm this? I just want to make sure I understand.
Flags: needinfo?(wmccloskey) → needinfo?(jmaher)
Hi Joel,
I did a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1591bcba900b
It has my changes from bug 1279086. Then, on top of that, I backed out bug 1302727. I consistently see a tart score of ~7. That's back to the old value from before any of these patches landed. So I think the regression is entirely caused by bug 1302727. Please close this bug if you agree.
(Reporter)

Comment 6

a year ago
:billm, thank you for checking on this.  I apologize that this was a randomization in your typical work flow.  It does seem as though bug 1302727 is the root cause.  As you look at the original graph on mozilla-inbound we had what looked like two regressions, even though there was a few revisions and retriggers.

I suspect bug 1302727 caused the regression, but bug 1279086 allowed it to fully be seen- fixing bug 1302727 would fix "both" changes in the graph.
Status: NEW → RESOLVED
Last Resolved: a year ago
Flags: needinfo?(jmaher)
Resolution: --- → DUPLICATE
Duplicate of bug: 1302727
Mark 51 fixed as bug 1302727 was fixed in 51.
status-firefox51: affected → fixed
Setting status the same as the dupe for 52.
status-firefox52: affected → fixed
You need to log in before you can comment on or make changes to this bug.