1362678 - 2.42 - 3.51% damp (linux64, windows7-32) regression on push a9339d769d2b03f5a21b8e7a804a0d2d528af1af (Fri May 5 2017)

Status: RESOLVED WONTFIX

(DevTools :: Netmonitor, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from push a9339d769d2b03f5a21b8e7a804a0d2d528af1af. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  4%  damp summary windows7-32 opt      308.41 -> 319.23
  2%  damp summary linux64 opt          301.08 -> 308.36


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=6413

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Looking at the code which landed and the subtests in the compare view:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=86de1de2f3f39bc225985f182ab992c97648f8a1&newProject=autoland&newRevision=a9339d769d2b03f5a21b8e7a804a0d2d528af1af&originalSignature=36adf201cb660409c00544b0a576c6f86c3eb669&newSignature=36adf201cb660409c00544b0a576c6f86c3eb669&framework=1

it looks very much related!

:rickychien, can you take a look at this regression and help determine if we can fix this, should back it out, or should accept it and move on?

Comment 2

[Ricky Chien [:rickychien] (inactive)]

Thanks for reporting this regression! I think it's ok to move on and I can take a look in these days.

Comment 3

[Ricky Chien [:rickychien] (inactive)]

Could you tell me which talos test I should run? I cannot figure it from https://treeherder.mozilla.org/perf.html#/alerts?id=6413.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

if you are pushing to try, then you need |./mach try -b o -p win32,linux64 -u none -t g2-e10s|.

locally it would be: |./mach talos-test -a g2|


As a note, we are not running non-e10s talos tests as that is our target for Firefox 57 (e10s only) and we need to focus our available machines on e10s performance.  This looks to be a non-e10s only regression- ideally we should make sure there are no e10s issues and then call this good.

Comment 5

[Ricky Chien [:rickychien] (inactive)]

Sorry I'm still not clear. According to https://treeherder.mozilla.org/perf.html#/alerts?id=6413, the damp regression looks like to happen on both e10s and non-e10s modes.

And |./mach talos-test -a g2| didin't work for me and threw:

FATAL - Test name is missing or invalid
FATAL - Running post_fatal callback...
FATAL - Exiting -1

when running it locally.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

oh, locally needs to be: |./mach talos-test -a damp|

I really had a bad typo earlier!  I looked at the data in the bug, not what alerts came in after filing the bug, so linux64 seems to be the platform where we have a flagged e10s damp regression.

Comment 7

[Ricky Chien [:rickychien] (inactive)]

Below is the reason why I'm confident that recent damp regression is not a high risk, and in fact that it has getting better since 90 days ago.

Perfheder result last 90 days

linux64 pgo

https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-inbound,b00d2367fbfcad44ac07235c55297cc427c570d4,1,1%5D&series=%5Bautoland,b00d2367fbfcad44ac07235c55297cc427c570d4,1,1%5D

linux64

https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bautoland,b97a8a540045b6139d2ce67b9c8b9a2f5c2918de,1,1%5D&series=%5Bmozilla-inbound,b97a8a540045b6139d2ce67b9c8b9a2f5c2918de,1,1%5D

Win32

https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bmozilla-inbound,36adf201cb660409c00544b0a576c6f86c3eb669,1,1%5D&series=%5Bautoland,36adf201cb660409c00544b0a576c6f86c3eb669,1,1%5D

Joel Maher, what do you think? Do you still see there are other potential risks? I'd suggest we can close the bug if we agree with that. Thanks!

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I agree that the overall trend is great here.  If you feel the regression is acceptable to take, then lets take it and move on!  It is great to see the big picture and also good to understand small details.

Comment 9

[Ricky Chien [:rickychien] (inactive)]

Thanks for your agreement. Let's take it and move on!

Comment 10

[Ryan VanderMeulen [:RyanVM]]

WONTFIX seems more appropriate since we're choosing to accept the regression.