1171653 - 4% Linux*/Win7 tp5o regression on Firefox e10s on June 01, 2015 from push baa9c64fea6f

Status: RESOLVED WONTFIX

(Testing :: Talos, defect, P5)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Talos has detected a Firefox performance regression from your commit baa9c64fea6f.  We need you to address this regression.

This is a list of all known regressions and improvements related to your bug:
http://alertmanager.allizom.org:8080/alerts.html?rev=baa9c64fea6f&showAll=1

On the page above you can see Talos alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test, please see: https://wiki.mozilla.org/Buildbot/Talos/Tests#tsvg.2C_tsvgx

Making a decision:
As the patch author we need your feedback to help us handle this regression.
*** Please let us know your plans by Monday, or the offending patch will be backed out! ***

Our wiki page outlines the common responses and expectations:
https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

our first e10s only regression!  yay for talos.

the problem is this is on m-c only and a large merge range.  We can't run talos e10s on try really (we can hack it).

not sure what route to take here.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

here is a graph showing the differences:
http://graphs.mozilla.org/graph.html#tests=[[255,1,47],[255,1,25],[255,1,35],[255,1,41],[255,1,43],[255,1,33]]&sel=1430855953902,1433447953903&displayrange=30&datatype=geo

Comment 3

[Mike Conley (:mconley) (:⚙️)]

This was a merge from fx-team. Do we a similar bump there?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we only run e10s talos on m-c (not fx-team or inbound).  and this is the *first* regression that is e10s only !!!

Comment 5

[Mike Conley (:mconley) (:⚙️)]

Ugh. Ok.

How possible would it be to try to backfill the missing fx-team e10s talos data from around that date?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

we don't have a way to run the tests on fx-team- maybe we can add that in and just not schedule it (i.e. create the builders).

:catlee, can we create builders for talos e10s jobs on fx-team (and ideally mozilla-inbound).  Then when we find an issue we can backfill.  Do let me know how this might work.

Comment 7

[Chris AtLee [:catlee]]

hm, I don't think we have a way right now to create the tests and not run them.

could we use seta for this?

Comment 8

[Mike Conley (:mconley) (:⚙️)]

There's not much we can do here until we have a changeset. :(

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

:catlee, seta doesn't work on Talos at the moment, if it did, then we could  apply the same logic of SETA to talos on inbound/fx-team for e10s and reduce resources.  Maybe it makes sense to do this.

kmoir, can you weigh in on how much work it might be to apply SETA to the talos builders?

Comment 10

[Kim Moir [:kmoir] ET]

Well, we would have to change the talos scheduler to use the class that looks at the skipconfig data. And we would need talos data generated by your SETA scripts so we could consume it.  So it does require significant testing like the previous implementation. However, we do have the code that works for opt and debug tests so the work should be more on the testing side, implementation shouldn't be that difficult in theory.

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Is tying this into our current automation practical, given that we'll probably only need to do this once?

I seem to recall MattN had some scripts that let us do some backfilling of talos data when we were working on Australis... MattN, are those scripts still around? Perhaps we could modify them for our purposes.

Comment 12

[Matthew N. [:MattN]]

http://hg.mozilla.org/users/mozilla_noorenberghe.ca/talos-tart/file/b53e872a557f/tart-nightlies.sh
http://hg.mozilla.org/users/mozilla_noorenberghe.ca/talos-tart/file/b53e872a557f/README-TART

A patch to moznightly was also needed so it didn't delete the downloaded build. I believe moznightly is gone now though there is a github issue to bring it back.

I'm on my phone on a plane now so these instructions aren't thorough but it may get you started. I can help next week at Whistler. I will need to add your machine ID to my server IIRC if you want to post there like the one script does. If you save the result files you can also POST them with later.

Comment 13

[Mike Conley (:mconley) (:⚙️)]

Alright, I don't think it makes much sense to get a bunch of releng or ateam people hacking on making this happen, since this is probably one-time-only.

I've requested a Linux talos machine. My plan is to write a script that will run the tp5o test on the machine for each push to fx-team within the regression range, and report the results.

jmaher - is it possible / advisable for me to have talos report the results from this machine to graph server for analysis? Or should I do the old trick of posting the results to a Google Spreadsheet or into a file for manual analysis?

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

it would be just fine to report to graph server as long as you have the branch and machine names correct.

Comment 15

[Mike Conley (:mconley) (:⚙️)]

So the fastest path (at least while at Whistler) seemed to be bisecting with try pushes between sessions.

Just a reminder, this is the regression range that was identified: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2c815cc65cc9&tochange=8707d35414f4


8707d35414f4: Bad

70a376c0f23d: Bad

dc2e19e737c7: Bad

d5adc9e191d7: Bad

09b27d21c789: Bad

507b6aba4555: Bad

d551aa12ebb1: Good

1a955124eccc: Good

9aed76a4ee0b: Good

2c815cc65cc9: Good

My bisection leads me to believe that this was caused by bug 1148582.

Comment 16

[Mike Conley (:mconley) (:⚙️)]

This was marked m7 so we could at least identify the regressing changeset. I think we've done that now.

Comment 17

[Timothy Nikkel (:tnikkel)]

I'm guessing this has the same cause as bug 1169756. Pushing

https://hg.mozilla.org/try/rev/20272d58e2e5

to try with whatever other options are needed to reproduce this could confirm.

Comment 18

[Mike Conley (:mconley) (:⚙️)]

There's a patch to force e10s enabled for talos by pointing it at an alternative talos repo / revision. That's the best we can do until bug 1174780 is fixed.

I'll do the try push comparison:

Before tn's patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=84968faa49e8

After: https://treeherder.mozilla.org/#/jobs?repo=try&revision=183f5d88ea27

Comment 19

[Mike Conley (:mconley) (:⚙️)]

Retriggers still coming in, but the initial results of this patch are compelling:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=84968faa49e8&newProject=try&newRevision=183f5d88ea27

Comment 20

[Mike Conley (:mconley) (:⚙️)]

Retriggers in - we have a winner!

Comment 21

[Timothy Nikkel (:tnikkel)]

Unfortunately landing that patch would be a correctness regression. See bug 1169756 comment 16 for example.

Comment 22

[Mike Conley (:mconley) (:⚙️)]

Hrm. Well, at least we know this is where the bottleneck is. Let me know if / when you've got another patch you'd like to test.

Comment 23

[Brad Lassey [:blassey] (use needinfo?)]

[Tracking Requested - why for this release]: regression in 41

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

from:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=aad95360a002&exclusion_profile=false&filter-searchStr=talos

here are the results:

Linux:
:( tp5o                 274.8 +/-  0% (7#)  [ +25.9%]     345.9 +/-  2% (7#)
Linux64:
:( tp5o                 247.0 +/-  1% (7#)  [ +28.6%]     317.6 +/-  1% (7#)
Win7:
:( tp5o                 229.7 +/-  0% (7#)  [  +9.1%]     250.7 +/-  0% (7#)
WinXP:
:( tp5o                 212.7 +/-  1% (7#)  [ +12.4%]     239.0 +/-  0% (6#)
Win8:
:( tp5o                 215.1 +/-  0% (7#)  [  +9.5%]     235.6 +/-  2% (7#)
OSX 10.10:
:( tp5o                 280.6 +/-  4% (7#)  [ +20.6%]     338.5 +/-  5% (7#)


notice that OSX 10.10 has a 20% hit, the rest are roughly the same.

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

sorry, wrong bug

Comment 26

[Kate Glazko]

Tracking in 41 because Comment 23

Comment 27

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

FF41 does not have e10s enabled by default. Moved tracking to 42 and 43 to ensure this gets attention there.

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

It doesn't look like it'll be useful to track this any more; I'd like to know though, how we will be testing and prioritizing performance issues when e10s is turned on. From talking with joel it sounds like we have e10s tests turned on for all pushes for talos now and so it will be easier to pinpoint future regressions. 

Should we close this, or is it still useful to leave it open?  Brad,  what do you think?

Comment 29

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #28)
> It doesn't look like it'll be useful to track this any more; I'd like to
> know though, how we will be testing and prioritizing performance issues when
> e10s is turned on. From talking with joel it sounds like we have e10s tests
> turned on for all pushes for talos now and so it will be easier to pinpoint
> future regressions. 
> 
> Should we close this, or is it still useful to leave it open?  Brad,  what
> do you think?

IMO, regressions should track the release they regressed in. If we are saying we don't care to fix this regression then close it as won't fix.

Comment 30

[Liz Henry (:lizzard) (relman/hg->git project)]

tnikkel, it may be up to you then. I asked Brad before I noticed you were assigned to the bug. Improving performance would be great of course, and I don't want to close this if you're still intending to work on it.