713055 - roll out mozharness desktop talos to mozilla-central + project branches when ready

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jeff Hammel]

[tracking bug]

Goal: get Talos on Mozharness in production

(Secondary goal: generally make Talos nicer to use and more maintainable)

Hard Blockers [P1]:

* Bug 650887 - desktop talos runner in mozharness; This is actually
  checked in and "works", though as Aki notes there may need to be
  more improvements a la configuration for a production setup. We
  should get Aki to ticket what else needs to be done. But 650887 is
  blocked by Bug 700722

* Bug 650890 - port remote talos to mozharness; Aki is working on
  this, but he's been busy.  Not sure what the status is.  See also
  bug 713003

* Bug 701506 - create python package webserver; If we're using
  mozharness to install talos into a virtualenv, we will need packages
  from somewhere.

* But 700722 - Talos process checking is over-ambitious and wrong; The
  short story: running `python talos_mozharness.py --appname
  /some/path/firefox` will err out since the talos subprocess will
  detect this command as a running firefox.  The patch there does this
  much better, but breaks on Mac, presumedly because it is looking for
  the wrong thing (maybe firefox.app?).  I haven't had a chance to
  investigate what it wants to find, but it is reproducible in
  staging.  All that is needed is time to tackle this.

* Bug 713003 - combine desktop talos mozharness and remote talos
  mozharness; strictly speaking, this isn't blocking but Aki wants it
  and its probably a good idea

* Bug 713017 - get buildbotcustom to use mozharness + production talos;
  Probably a releng task. Not sure who should head up this effort.
  It is blocked by all of the above (mostly). Of course, this will all
  have to be heavily staged as well.

* Bug 694625 - talos should consume mozprocess; not strictly necessary
  but would be a substantive improvement.

Soft Blockers [P2]:

None of these are strictly necessary, but they are nice improvements
to take as part of this effort and most of them are low-hanging fruit.

* Bug 705809 - Talos should not depend on scripts being run from the
  talos directory; See this magical workaround:
  http://hg.mozilla.org/build/mozharness/file/be92a9addbd2/scripts/talos_script.py#l91

* Bug 705811 - run_tests.py should become a console_script;
  This goes for PerfConfigurator, etc, too.

* install story: currently we pretend to install pageloader as part of
  setup.py.  However, we don't extract it correctly :( We need to
  repack as there is a top level of namespace we don't need (bug
  709340).  However, we also have an install script
  (http://hg.mozilla.org/build/talos/file/cb1b7a64f98e/INSTALL.py
  ). Maybe that is a better place to do this?  It is probably worth
  figuring out what to do with the install script.  Currently it
  clones a whole new talos, but it might be worth setting up in place
  if it is already downloaded/checked out.  See also bug 701490,
  pageloader.xpi does not get installed from easy_install

* Bug 709349 - Getting broken pipe running talos in --develop mode; we
  probably don't want these showing up in production logs.

* Bug 694638 - talos should consume mozprofile

Also, talos.zip :
https://bugzilla.mozilla.org/show_bug.cgi?id=707218 . We should do
something about that

Comment 1

[Aki Sasaki (not active)]

(In reply to Jeff Hammel [:jhammel] from comment #0)
> * Bug 650890 - port remote talos to mozharness; Aki is working on
>   this, but he's been busy.  Not sure what the status is.  See also
>   bug 713003

If this bug is tracking the Q2 goals, this isn't a hard blocker.
The Q2 goal for remote talos is a working poc only, not running in production.

> * Bug 713017 - get buildbotcustom to use mozharness + production talos;
>   Probably a releng task. Not sure who should head up this effort.
>   It is blocked by all of the above (mostly). Of course, this will all
>   have to be heavily staged as well.

This will use the same code paths as peptest + Jordan's unit tests.  I imagine we'll run in parallel with non-mozharness talos for a while.
We'll need config files for talos before buildbot code changes.

Comment 2

[Jeff Hammel]

I'm removing the "get talos on mozbase" dependencies from this bug.  While we want to do this (soon), this bug is scheduled to be done by EOQ2 and these are not in scope for that

Comment 3

[Jeff Hammel]

One month left before this is supposed to be "done".  Triaging says:

* bug 650890 - port remote talos to mozharness: not a strict blocker; assigned to Callek

* bug 701506 - create python package webserver: not sure what the status is at the moment and it is unowned, though it looks like most of what was desired by RelEng is basically done? We'll have to file bugs to get the packages Talos requires up after this is in a finished state

* bug 713017 - get buildbotcustom to use mozharness + talos: assigned to :aki, not sure what the status is

* bug 760320 - production config files for mozharness Talos : unassigned, filed today

Is there anything else I'm missing here?  The unassigned bugs should probably find owners.  Not sure if we need any changes talos or mozharness side, but if so I could take those or otherwise assist at getting any of this done

Comment 4

[Aki Sasaki (not active)]

(In reply to Jeff Hammel [:jhammel] from comment #3)
> * bug 760320 - production config files for mozharness Talos : unassigned,
> filed today

Imo, this is part of the mozharness script.
I can help you understand mozharness config files if needed.

Comment 5

[cmtalbert]

Aki what is the status on bug 713017?

Comment 6

[Aki Sasaki (not active)]

I just picked it up on Wednesday and haven't started it.
It will require config files for Talos, and we're blocked on pushing that to production til after we have the python package server.

I have buildduty + a release next week, so I probably won't get to spend much time on it til the week of the 11th.

Comment 7

[Jeff Hammel]

adding bug 764592 as a blocker.  This is probably not strictly true but close enough.  We'll want to go into this at least knowing our versioning story wrt talos + m-c

Comment 8

[Jeff Hammel]

Last Weds (Dec 19, 2012) we had a mozharness meeting where we audited the status of this bug.  We're still hammering out dependencies.  I'll give a A*Team POV on what else needs to be done there:

* bug 795236 : have fixed 822478 which will allow --authfile to use URLs, not just file paths; we still need config file changes which involves someone who knows where the authfile actually lives, so probably releng

* bug 795531 : being worked on by releng

* bug 802801 : someone needs to look at this; jmaher assigned ;

* bug 804385 : this probably shouldn't block deployment if its rare

* bug 805925 :
** we need to upload new talos + other packages: bug 805925
** So this still won't fix the problem.  But it will put us on parity with what is happening in production

* bug 812609 : releng is working on this

Comment 9

[Aki Sasaki (not active)]

Blocking on bug 837022's env update.

Comment 10

[Aki Sasaki (not active)]

Landed a followup fix https://hg.mozilla.org/build/mozharness/rev/19e86b400db1 to get unit.sh passing again.

Comment 11

[Armen [:armenzg]]

I've hidden the rev3 machines from Cedar so we can focus:
https://tbpl.mozilla.org/?tree=Cedar&jobname=talos

Did someone from the a-team say that can help you verify the talos numbers?
I don't know what the right tools is these days.

Is the minidump bug still happening?
Perhaps you can ask ted or the a-team if they have a patch that can ensure crashing so we can do a try push.

I think our next branch to target should be try rather than mozilla-inbound and/or mozilla-central.

Comment 12

[Armen [:armenzg]]

Attachment: [checked-in] talos.json changes

Should we be landing this patch to mozilla-inbound now-ish?
That way we don't have problems when we switch to talos mozharness.

Comment 13

[Jason Yeo[:jyeo]]

Attachment: enable talos on moz-inbound

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 774741 [details] [diff] [review]
enable talos on moz-inbound

Let's roll out to try first, then all m-c level branches next.
I don't think it makes sense to only roll out to inbound.

Comment 15

[Armen [:armenzg]]

Comment on attachment 774674 [details] [diff] [review]
[checked-in] talos.json changes

https://hg.mozilla.org/releases/mozilla-aurora/rev/b1048f6c61aa
https://hg.mozilla.org/releases/mozilla-beta/rev/a4eb39196335
https://hg.mozilla.org/releases/mozilla-release/rev/82c75b411921

Comment 16

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/3d1c2ca7efe8

Comment 17

[Jason Yeo[:jyeo]]

Attachment: [checked-in] enable talos on try

Comment 18

[Armen [:armenzg]]

Comment on attachment 775683 [details] [diff] [review]
[checked-in] enable talos on try

Review of attachment 775683 [details] [diff] [review]:
-----------------------------------------------------------------

Let's talk first before landing it.

Comment 19

[Armen [:armenzg]]

Comment on attachment 775683 [details] [diff] [review]
[checked-in] enable talos on try

https://hg.mozilla.org/build/buildbot-configs/rev/bacbba917d32

Comment 20

[Armen [:armenzg]]

Merged to the production and live on the try server.
https://tbpl.mozilla.org/?tree=Try&jobname=talos

We will have a look for few days and then enable it across the board.

Comment 21

[Armen [:armenzg]]

Attachment: only consider return codes for talos mozharness

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 778586 [details] [diff] [review]
only consider return codes for talos mozharness

Before we land this and reconfig, we should verify the talos script actually exits with the appropriate self.return_code.

To do that we may have to add more self.buildbot_status() calls in mozharness.mozilla.testing.talos.  However, iirc, talos never goes orange, only green/red/retry.

Comment 23

[Ed Morley [:emorley]]

(In reply to Aki Sasaki [:aki] from comment #22)
> However, iirc, talos never goes orange,
> only green/red/retry.

Talos also goes orange for crashes and test-unexpected-fails as of bug 829728, via:
https://hg.mozilla.org/build/buildbotcustom/file/f24d9219c221/steps/talos.py#l115

Comment 24

[Armen [:armenzg]]

Moved the talos return codes to bug 896015.

Comment 25

[Armen [:armenzg]]

Attachment: talos_mozharness.diff

Enable talos mozharness on all FF25 branches.

Comment 26

[Armen [:armenzg]]

Attachment: different_builders.diff

This shows which builders are being modified

Comment 27

[Armen [:armenzg]]

Attachment: [mc] Match talos.zip to talos_revision

Comment 28

[Armen [:armenzg]]

Attachment: [ma] Match talos.zip to talos_revision and use talos_repo

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779924 [details] [diff] [review]
[mc] Match talos.zip to talos_revision

Review of attachment 779924 [details] [diff] [review]:
-----------------------------------------------------------------

woot!

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779927 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/talos/talos.json
@@ +4,5 @@
>          "path": ""
>      },
>      "global": {
> +        "talos_repo": "http://hg.mozilla.org/build/talos",
> +        "talos_revision": "a11542b55a70"

the revision is wrong here

Comment 31

[Armen [:armenzg]]

Attachment: [mb] Match talos.zip to talos_revision and use talos_repo

Comment 32

[Armen [:armenzg]]

Attachment: [mr] Match talos.zip to talos_revision and use talos_repo

Comment 33

[Armen [:armenzg]]

Thanks for catching the mismatched revision.

Even though we're *not* enabling the new talos on older branches and will ride the trains (I think we can revisit this if we want to), I want talos.json to look the right way.

Comment 34

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779933 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779933 [details] [diff] [review]:
-----------------------------------------------------------------

this is not a valid revision :)

Comment 35

[Armen [:armenzg]]

Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

This one seems correct:
http://hg.mozilla.org/build/talos/rev/a11542b55a70

Comment 36

[Armen [:armenzg]]

Attachment: [mr] Match talos.zip to talos_revision and use talos_repo

I have no idea what that was about. Sorry!

http://hg.mozilla.org/build/talos/rev/560806cfa208

Comment 37

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779927 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/talos/talos.json
@@ +4,5 @@
>          "path": ""
>      },
>      "global": {
> +        "talos_repo": "http://hg.mozilla.org/build/talos",
> +        "talos_revision": "a11542b55a70"

ok, this is the right revision for the right branch.

Comment 38

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779935 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779935 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!

Comment 39

[Armen [:armenzg]]

I think we're going to have to postpone enabling this tomorrow since I would not like to start using an incorrect talos_revision and possibly cause regressions for using an older talos versions.
It's unfortunate but I think it is the safest.
I will have to keep an eye on newer talos.zip revision requests.

Anyone objects? I was thinking of pushing it to Monday.

On another note, can I land a patch on mozilla-central without waiting for a merge from m-i? or would I cause a merge conflict?

https://hg.mozilla.org/integration/mozilla-inbound/rev/0d4ab37e3f3e
https://hg.mozilla.org/releases/mozilla-aurora/rev/ac9b464cb3c8
https://hg.mozilla.org/releases/mozilla-beta/rev/64046aafb054
https://hg.mozilla.org/releases/mozilla-release/rev/481145f83cc6

Comment 40

[Armen [:armenzg]]

Attachment: update talos.json to match values from config.py

I'm waiting on jyeo to let me know if we're making use of "suites" inside of talos.json yet.

If we're, then we have to wait for this patch to spread across the branches.

Comment 41

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779958 [details] [diff] [review]
update talos.json to match values from config.py

Review of attachment 779958 [details] [diff] [review]:
-----------------------------------------------------------------

update buildbot-configs first, I backed out some stuff, sorry.

::: testing/talos/talos.json
@@ +63,5 @@
> +                "ignore_first:5",
> +                "--filter",
> +                "median"
> +            ]
> +        },

we can delete both of these: otherx, and svgx.

@@ +74,5 @@
>                  "median"
>              ]
>          },
> +        "rafx": {
> +            "tests": ["tcanvasmark"],

this should have tsvgx, tscrollx

Comment 42

[Armen [:armenzg]]

Attachment: update talos.json to match values from config.py

Comment 43

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 779975 [details] [diff] [review]
update talos.json to match values from config.py

Review of attachment 779975 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great!

Comment 44

[Armen [:armenzg]]

Comment on attachment 779975 [details] [diff] [review]
update talos.json to match values from config.py

I've landed it without removing the talos.zip block as that would have been bad:
https://hg.mozilla.org/integration/mozilla-inbound/rev/496a7582cf9e

Comment 45

[Armen [:armenzg]]

Testing needed before we can enable talos mozharness across the board:
https://tbpl.mozilla.org/?tree=Try&rev=b5feca0c0c50
https://hg.mozilla.org/projects/cedar/rev/7cfd536a3d8a

Comment 46

[John Hopkins (:jhopkins)]

Comment on attachment 779916 [details] [diff] [review]
talos_mozharness.diff

As discussed on IRC:
armenzg: try, cedar and ash should not change
jhopkins: "Rev4 MacOSX Lion 10.7 try talos" seems to have changed

I was comparing dump_master.py output before and after the patch and noticed the changed builder factories (note that 'try opt test' is unchanged but 'try talos' does indeed use a different factory than before):

< Rev4 MacOSX Lion 10.7 try opt test mochitest-browser-chrome ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test mochitest-browser-chrome ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test mochitest-other ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test mochitest-other ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test reftest-ipc ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test reftest-ipc ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test reftest ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test reftest ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test xpcshell ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test xpcshell ScriptFactory
< Rev4 MacOSX Lion 10.7 try talos chromez ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos chromez TalosFactory
< Rev4 MacOSX Lion 10.7 try talos dirtypaint ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos dirtypaint TalosFactory
< Rev4 MacOSX Lion 10.7 try talos dromaeojs ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos dromaeojs TalosFactory
< Rev4 MacOSX Lion 10.7 try talos other ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos other TalosFactory
< Rev4 MacOSX Lion 10.7 try talos svgr ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos svgr TalosFactory
< Rev4 MacOSX Lion 10.7 try talos tp5o ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos tp5o TalosFactory

Comment 47

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0d4ab37e3f3e

Comment 48

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/496a7582cf9e

Comment 49

[Armen [:armenzg]]

Talos.zip is needed for mobile. Re-testing:
https://tbpl.mozilla.org/?tree=Cedar

Status summary:
###############
* I need to provide a patch for Monday to enable talos mozharnes for all FF25 trees
* I want to make sure that the talos.json modifications from comment 44 do not cause any regressions on Monday
* I want to make sure that talos.json is consistent across older branches in case a developer wants to push a change to the try server (since it will be using talos mozharness).

[1] https://hg.mozilla.org/integration/mozilla-inbound/rev/496a7582cf9e

Comment 50

[Armen [:armenzg]]

> * I want to make sure that talos.json is consistent across older branches in
> case a developer wants to push a change to the try server (since it will be
> using talos mozharness).

https://hg.mozilla.org/releases/mozilla-aurora/rev/729725cf05b8
https://hg.mozilla.org/releases/mozilla-beta/rev/d78022c9d69c
https://hg.mozilla.org/releases/mozilla-release/rev/2788eff35d00
https://hg.mozilla.org/releases/mozilla-b2g18/rev/7d759454371c
https://hg.mozilla.org/releases/mozilla-esr17/rev/c6d465f029f4
https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/c933e9b80897

Comment 51

[Armen [:armenzg]]

Attachment: enable talos mozharness for all FF25 trees

Only one line has been added.

Comment 52

[Armen [:armenzg]]

I'm looking at this http://perf.snarkfest.net/compare-talos/index.html?oldRevs=f479167e92d2&newRev=7cfd536a3d8a&submit=true where I'm comparing two different revisions of Cedar I noticed a huge regression in tp5n_main_normal_fileio_paint but I can't find which job on Cedar runs it.
Any ideas?

https://tbpl.mozilla.org/?tree=Cedar&jobname=talos&showall=1

Comment 53

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure how to use the compare-talos toolchain, but we could look at some revisions on cedar and see what the raw values are.  This looks to be related to xperf, and those values are subject to change a lot.

Comment 54

[Armen [:armenzg]]

This is tedious.
It would be great that compare-talos allowed to create graphs URLs to compare to other branches or within the same branch (that is what I do in the following links).

From:
http://perf.snarkfest.net/compare-talos/index.html?oldRevs=f479167e92d2&newRev=7cfd536a3d8a&submit=true

I've built the following URLs:
http://graphs.mozilla.org/graph.html#tests=[[224,63,24],[224,26,24]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[251,26,25],[251,63,25]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[244,26,25],[244,63,25]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[245,26,25],[245,63,25]]&sel=none&displayrange=7&datatype=running
I would say that we're fine and it is due to noisy talos jobs.

Unless someone strongly believes that I should keep up posting URL for each regressions mentioned on compare-talos for more than 3% deviance I would like to assume that we're fine.

I'm facing the same situation as to went I tried if talos-mozharness causes clear regressions compared to buildbotcustom/factory.py. It requires lots of eye analysis.

Missing URLs for these regressions:
- Ts Paint, MED Dirty Profile (tspaint_places_generated_med) - Mac 10.8 
- Ts Paint, MAX Dirty Profile (tspaint_places_generated_max) - Mac 10.8 & Ub. 32 
- TResize (tresize) - Win8
- Tp5 Optimized Responsiveness (tp5o_responsiveness_paint) - Mac 10.8, Win7 & Win8
- Tp5 Optimized MozAfterPaint (tp5o_shutdown_paint) - all but win7 & win8
- Tp5 Optimized (Modified Page List Bytes) (tp5o_modlistbytes_paint) - Win7

Comment 55

[Armen [:armenzg]]

We'll deploy this tomorrow morning.
Review is expected to come through today.

Comment 56

[John Hopkins (:jhopkins)]

Comment on attachment 781042 [details] [diff] [review]
enable talos mozharness for all FF25 trees

armenzg: can you please post an updated patch?  This one no longer applies cleanly to buildbot-configs.

Comment 57

[Armen [:armenzg]]

Attachment: enable talos mozharness for all FF25 trees (updated)

Comment 58

[Armen [:armenzg]]

Attachment: Results from firt changeset when we enabled talos mozharness on Cedar: 14e3e9ab9994

Comment 59

[Armen [:armenzg]]

Attachment: Results from talos.json updates on Cedar: 7cfd536a3d8a

Comment 60

[Armen [:armenzg]]

I don't really have a clear way to determine what suites will clearly be affected by the talos mozharness tomorrow.

The last two attachments were obtained with this:
python talos/compare.py --revision=14e3e9ab9994 --branch=Cedar --masterbranch=Firefox --print-graph-url
python talos/compare.py --revision=14e3e9ab9994 --branch=Cedar --masterbranch=Firefox --print-graph-url

I wanted to understand things a little better and be sure that I know exactly what is going to change so I can answer any questions from developers. I wanted to be more educated in a way.

For now, I will re-paste the analysis that jmaher did a bit ago and hope that nothing new has cropped in.

From bug 802801:

(In reply to Joel Maher (:jmaher) from comment #10)
> So it appears that our ts test (start/stop the browser 20 times) is
> problematic.  the places_med|max are just different profiles used while
> running the test.  I have verified this on a few different changesets.  
> 
> I really don't understand how changing to mozharness could cause this, but
> maybe there is some additional overhead induced on the system with
> mozharness when it comes to launching a new process.  
> 
> We could take this as a bump in the numbers and accept that.  I am open to
> any thoughts here.

Comment 61

[Armen [:armenzg]]

This is live now.

Comment 62

[Armen [:armenzg]]

This is probably one of the expected regressions to be reported:
http://graphs.mozilla.org/graph.html#tests=[[226,94,25],[227,94,25],[226,131,25],[227,131,25]]&sel=1375122199529,1375198347356&displayrange=7&datatype=running

Comment 63

[Aki Sasaki (not active)]

I'm going to guess we're done here... ?
I don't think the two existing blocker bugs are actually strictly blocking.

Comment 64

[Jonathan Griffin (:jgriffin)]

I agree; I think this can be closed.