6.16% tart (windows7-32) regression on push ea326f6e7e6c (Sat Sep 17 2016)




2 years ago
2 years ago


(Reporter: jmaher, Assigned: billm)


({perf, regression, talos-regression})

51 Branch
perf, regression, talos-regression
Dependency tree / graph

Firefox Tracking Flags

(firefox49 unaffected, firefox50 unaffected, firefox51 fixed, firefox52 fixed)




2 years ago
Talos has detected a Firefox performance regression from push eaf5eb6f8fa. As author of one of the patches included in that push, we need your help to address this regression.


  6%  tart summary windows7-32 opt e10s     7.49 -> 7.95

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=3295

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

2 years ago
:billm, I see you are the patch author here can you take a look at this regression and help find the root cause.  This was landed right before we uplifted on Monday (and right after another regression, so we had missed it)
Flags: needinfo?(wmccloskey)


2 years ago
Summary: 6.16% tart (windows7-32) regression on push eaf5eb6f8fa (Sat Sep 17 2016) → 6.16% tart (windows7-32) regression on push ea326f6e7e6c (Sat Sep 17 2016)

Comment 2

2 years ago
and I filed the bug on the downstream alert, here is the proper alert we should worry about for tracking in perfherder:
== Change summary for alert #3322 (as of September 21 2016 03:19 UTC) ==


 10%  tart summary windows7-32 opt e10s     7.11 -> 7.86

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=3322
Assignee: nobody → wmccloskey
Flags: needinfo?(wmccloskey)
status-firefox49: --- → unaffected
status-firefox50: --- → unaffected
status-firefox51: --- → affected
status-firefox52: --- → affected

Comment 3

2 years ago
:billm, do you have any questions about this regression?  Any luck coming up with a fix?
Flags: needinfo?(wmccloskey)
Hi Joel,
I'm having some trouble understanding the magnitude of the regression here. Bug 1302727 landed shortly before my patch and it also regressed TART. I think the right comparison to make is this one:

since revision c6876f222943 landed after bug 1302727. I do see a 3.90% regression, but the confidence is "low".

I also looked at the graph on inbound:

It seems like bug 1302727 took us from ~6.9 to ~7.5. Then my patch took us to ~7.9. Can you confirm this? I just want to make sure I understand.
Flags: needinfo?(wmccloskey) → needinfo?(jmaher)
Hi Joel,
I did a try push:
It has my changes from bug 1279086. Then, on top of that, I backed out bug 1302727. I consistently see a tart score of ~7. That's back to the old value from before any of these patches landed. So I think the regression is entirely caused by bug 1302727. Please close this bug if you agree.

Comment 6

2 years ago
:billm, thank you for checking on this.  I apologize that this was a randomization in your typical work flow.  It does seem as though bug 1302727 is the root cause.  As you look at the original graph on mozilla-inbound we had what looked like two regressions, even though there was a few revisions and retriggers.

I suspect bug 1302727 caused the regression, but bug 1279086 allowed it to fully be seen- fixing bug 1302727 would fix "both" changes in the graph.
Last Resolved: 2 years ago
Flags: needinfo?(jmaher)
Resolution: --- → DUPLICATE
Duplicate of bug: 1302727
Mark 51 fixed as bug 1302727 was fixed in 51.
status-firefox51: affected → fixed
Setting status the same as the dupe for 52.
status-firefox52: affected → fixed
You need to log in before you can comment on or make changes to this bug.