Last Comment Bug 713055 - roll out mozharness desktop talos to mozilla-central + project branches when ready
: roll out mozharness desktop talos to mozilla-central + project branches when ...
Status: RESOLVED FIXED
[mozharness+talos][leave open]
:
Product: Release Engineering
Classification: Other
Component: Mozharness (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Jason Yeo[:jyeo]
: Chris AtLee [:catlee]
Mentors:
Depends on: 650887 700722 701490 701506 705809 705811 707218 709340 709349 713003 713017 760320 761809 766692 767042 794587 795236 795531 802801 803647 804385 805925 812609 812726 823306 837022 838485 853599 853679 855554 878572 880414 880876 891525 894980 895721 896015 899570 899784 899793 900015 900545 900605
Blocks: 734466 764588
  Show dependency treegraph
 
Reported: 2011-12-22 12:02 PST by Jeff Hammel
Modified: 2014-07-09 10:10 PDT (History)
13 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
[checked-in] talos.json changes (4.67 KB, patch)
2013-07-12 08:26 PDT, Armen Zambrano [:armenzg] - Engineering productivity
aki: review+
jasonyeo88: feedback+
armenzg: checked‑in+
Details | Diff | Review
enable talos on moz-inbound (819 bytes, patch)
2013-07-12 10:02 PDT, Jason Yeo[:jyeo]
no flags Details | Diff | Review
[checked-in] enable talos on try (1.50 KB, patch)
2013-07-15 08:56 PDT, Jason Yeo[:jyeo]
armenzg: review+
armenzg: checked‑in+
Details | Diff | Review
only consider return codes for talos mozharness (1.23 KB, patch)
2013-07-19 11:29 PDT, Armen Zambrano [:armenzg] - Engineering productivity
aki: review+
Details | Diff | Review
talos_mozharness.diff (11.33 KB, patch)
2013-07-23 11:44 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jhopkins: review-
Details | Diff | Review
different_builders.diff (71.57 KB, patch)
2013-07-23 11:45 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Review
[mc] Match talos.zip to talos_revision (660 bytes, patch)
2013-07-23 11:49 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review+
armenzg: checked‑in+
Details | Diff | Review
[ma] Match talos.zip to talos_revision and use talos_repo (773 bytes, patch)
2013-07-23 11:52 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review+
armenzg: checked‑in+
Details | Diff | Review
[mb] Match talos.zip to talos_revision and use talos_repo (773 bytes, patch)
2013-07-23 11:55 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review+
armenzg: checked‑in+
Details | Diff | Review
[mr] Match talos.zip to talos_revision and use talos_repo (3.18 KB, patch)
2013-07-23 11:55 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review-
Details | Diff | Review
[mr] Match talos.zip to talos_revision and use talos_repo (774 bytes, patch)
2013-07-23 12:08 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review+
armenzg: checked‑in+
Details | Diff | Review
update talos.json to match values from config.py (2.97 KB, patch)
2013-07-23 12:50 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review-
Details | Diff | Review
update talos.json to match values from config.py (2.43 KB, patch)
2013-07-23 13:14 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jmaher: review+
armenzg: checked‑in+
Details | Diff | Review
enable talos mozharness for all FF25 trees (12.17 KB, patch)
2013-07-25 09:27 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jhopkins: review-
Details | Diff | Review
enable talos mozharness for all FF25 trees (updated) (12.62 KB, patch)
2013-07-29 12:12 PDT, Armen Zambrano [:armenzg] - Engineering productivity
jhopkins: review+
armenzg: checked‑in+
Details | Diff | Review
Results from firt changeset when we enabled talos mozharness on Cedar: 14e3e9ab9994 (9.54 KB, text/plain)
2013-07-29 12:51 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details
Results from talos.json updates on Cedar: 7cfd536a3d8a (9.72 KB, text/plain)
2013-07-29 12:52 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details

Description Jeff Hammel 2011-12-22 12:02:18 PST
[tracking bug]

Goal: get Talos on Mozharness in production

(Secondary goal: generally make Talos nicer to use and more maintainable)

Hard Blockers [P1]:

* Bug 650887 - desktop talos runner in mozharness; This is actually
  checked in and "works", though as Aki notes there may need to be
  more improvements a la configuration for a production setup. We
  should get Aki to ticket what else needs to be done. But 650887 is
  blocked by Bug 700722

* Bug 650890 - port remote talos to mozharness; Aki is working on
  this, but he's been busy.  Not sure what the status is.  See also
  bug 713003

* Bug 701506 - create python package webserver; If we're using
  mozharness to install talos into a virtualenv, we will need packages
  from somewhere.

* But 700722 - Talos process checking is over-ambitious and wrong; The
  short story: running `python talos_mozharness.py --appname
  /some/path/firefox` will err out since the talos subprocess will
  detect this command as a running firefox.  The patch there does this
  much better, but breaks on Mac, presumedly because it is looking for
  the wrong thing (maybe firefox.app?).  I haven't had a chance to
  investigate what it wants to find, but it is reproducible in
  staging.  All that is needed is time to tackle this.

* Bug 713003 - combine desktop talos mozharness and remote talos
  mozharness; strictly speaking, this isn't blocking but Aki wants it
  and its probably a good idea

* Bug 713017 - get buildbotcustom to use mozharness + production talos;
  Probably a releng task. Not sure who should head up this effort.
  It is blocked by all of the above (mostly). Of course, this will all
  have to be heavily staged as well.

* Bug 694625 - talos should consume mozprocess; not strictly necessary
  but would be a substantive improvement.

Soft Blockers [P2]:

None of these are strictly necessary, but they are nice improvements
to take as part of this effort and most of them are low-hanging fruit.

* Bug 705809 - Talos should not depend on scripts being run from the
  talos directory; See this magical workaround:
  http://hg.mozilla.org/build/mozharness/file/be92a9addbd2/scripts/talos_script.py#l91

* Bug 705811 - run_tests.py should become a console_script;
  This goes for PerfConfigurator, etc, too.

* install story: currently we pretend to install pageloader as part of
  setup.py.  However, we don't extract it correctly :( We need to
  repack as there is a top level of namespace we don't need (bug
  709340).  However, we also have an install script
  (http://hg.mozilla.org/build/talos/file/cb1b7a64f98e/INSTALL.py
  ). Maybe that is a better place to do this?  It is probably worth
  figuring out what to do with the install script.  Currently it
  clones a whole new talos, but it might be worth setting up in place
  if it is already downloaded/checked out.  See also bug 701490,
  pageloader.xpi does not get installed from easy_install

* Bug 709349 - Getting broken pipe running talos in --develop mode; we
  probably don't want these showing up in production logs.

* Bug 694638 - talos should consume mozprofile

Also, talos.zip :
https://bugzilla.mozilla.org/show_bug.cgi?id=707218 . We should do
something about that
Comment 1 Aki Sasaki [:aki] 2012-05-30 13:30:38 PDT
(In reply to Jeff Hammel [:jhammel] from comment #0)
> * Bug 650890 - port remote talos to mozharness; Aki is working on
>   this, but he's been busy.  Not sure what the status is.  See also
>   bug 713003

If this bug is tracking the Q2 goals, this isn't a hard blocker.
The Q2 goal for remote talos is a working poc only, not running in production.

> * Bug 713017 - get buildbotcustom to use mozharness + production talos;
>   Probably a releng task. Not sure who should head up this effort.
>   It is blocked by all of the above (mostly). Of course, this will all
>   have to be heavily staged as well.

This will use the same code paths as peptest + Jordan's unit tests.  I imagine we'll run in parallel with non-mozharness talos for a while.
We'll need config files for talos before buildbot code changes.
Comment 2 Jeff Hammel 2012-05-31 16:16:49 PDT
I'm removing the "get talos on mozbase" dependencies from this bug.  While we want to do this (soon), this bug is scheduled to be done by EOQ2 and these are not in scope for that
Comment 3 Jeff Hammel 2012-05-31 17:03:44 PDT
One month left before this is supposed to be "done".  Triaging says:

* bug 650890 - port remote talos to mozharness: not a strict blocker; assigned to Callek

* bug 701506 - create python package webserver: not sure what the status is at the moment and it is unowned, though it looks like most of what was desired by RelEng is basically done? We'll have to file bugs to get the packages Talos requires up after this is in a finished state

* bug 713017 - get buildbotcustom to use mozharness + talos: assigned to :aki, not sure what the status is

* bug 760320 - production config files for mozharness Talos : unassigned, filed today

Is there anything else I'm missing here?  The unassigned bugs should probably find owners.  Not sure if we need any changes talos or mozharness side, but if so I could take those or otherwise assist at getting any of this done
Comment 4 Aki Sasaki [:aki] 2012-05-31 17:07:29 PDT
(In reply to Jeff Hammel [:jhammel] from comment #3)
> * bug 760320 - production config files for mozharness Talos : unassigned,
> filed today

Imo, this is part of the mozharness script.
I can help you understand mozharness config files if needed.
Comment 5 cmtalbert 2012-06-01 15:10:17 PDT
Aki what is the status on bug 713017?
Comment 6 Aki Sasaki [:aki] 2012-06-01 15:17:58 PDT
I just picked it up on Wednesday and haven't started it.
It will require config files for Talos, and we're blocked on pushing that to production til after we have the python package server.

I have buildduty + a release next week, so I probably won't get to spend much time on it til the week of the 11th.
Comment 7 Jeff Hammel 2012-06-13 14:35:55 PDT
adding bug 764592 as a blocker.  This is probably not strictly true but close enough.  We'll want to go into this at least knowing our versioning story wrt talos + m-c
Comment 8 Jeff Hammel 2012-12-26 12:38:57 PST
Last Weds (Dec 19, 2012) we had a mozharness meeting where we audited the status of this bug.  We're still hammering out dependencies.  I'll give a A*Team POV on what else needs to be done there:

* bug 795236 : have fixed 822478 which will allow --authfile to use URLs, not just file paths; we still need config file changes which involves someone who knows where the authfile actually lives, so probably releng

* bug 795531 : being worked on by releng

* bug 802801 : someone needs to look at this; jmaher assigned ;

* bug 804385 : this probably shouldn't block deployment if its rare

* bug 805925 :
** we need to upload new talos + other packages: bug 805925
** So this still won't fix the problem.  But it will put us on parity with what is happening in production

* bug 812609 : releng is working on this
Comment 9 Aki Sasaki [:aki] 2013-02-11 17:31:18 PST
Blocking on bug 837022's env update.
Comment 10 Aki Sasaki [:aki] 2013-06-10 14:30:20 PDT
Landed a followup fix https://hg.mozilla.org/build/mozharness/rev/19e86b400db1 to get unit.sh passing again.
Comment 11 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-09 13:35:26 PDT
I've hidden the rev3 machines from Cedar so we can focus:
https://tbpl.mozilla.org/?tree=Cedar&jobname=talos

Did someone from the a-team say that can help you verify the talos numbers?
I don't know what the right tools is these days.

Is the minidump bug still happening?
Perhaps you can ask ted or the a-team if they have a patch that can ensure crashing so we can do a try push.

I think our next branch to target should be try rather than mozilla-inbound and/or mozilla-central.
Comment 12 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-12 08:26:07 PDT
Created attachment 774674 [details] [diff] [review]
[checked-in] talos.json changes

Should we be landing this patch to mozilla-inbound now-ish?
That way we don't have problems when we switch to talos mozharness.
Comment 13 Jason Yeo[:jyeo] 2013-07-12 10:02:45 PDT
Created attachment 774741 [details] [diff] [review]
enable talos on moz-inbound
Comment 14 Aki Sasaki [:aki] 2013-07-12 10:12:22 PDT
Comment on attachment 774741 [details] [diff] [review]
enable talos on moz-inbound

Let's roll out to try first, then all m-c level branches next.
I don't think it makes sense to only roll out to inbound.
Comment 16 Ed Morley [:emorley] 2013-07-15 03:01:44 PDT
https://hg.mozilla.org/mozilla-central/rev/3d1c2ca7efe8
Comment 17 Jason Yeo[:jyeo] 2013-07-15 08:56:51 PDT
Created attachment 775683 [details] [diff] [review]
[checked-in] enable talos on try
Comment 18 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-15 10:05:02 PDT
Comment on attachment 775683 [details] [diff] [review]
[checked-in] enable talos on try

Review of attachment 775683 [details] [diff] [review]:
-----------------------------------------------------------------

Let's talk first before landing it.
Comment 19 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-16 05:57:17 PDT
Comment on attachment 775683 [details] [diff] [review]
[checked-in] enable talos on try

https://hg.mozilla.org/build/buildbot-configs/rev/bacbba917d32
Comment 20 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-16 10:50:45 PDT
Merged to the production and live on the try server.
https://tbpl.mozilla.org/?tree=Try&jobname=talos

We will have a look for few days and then enable it across the board.
Comment 21 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-19 11:29:24 PDT
Created attachment 778586 [details] [diff] [review]
only consider return codes for talos mozharness
Comment 22 Aki Sasaki [:aki] 2013-07-19 11:32:00 PDT
Comment on attachment 778586 [details] [diff] [review]
only consider return codes for talos mozharness

Before we land this and reconfig, we should verify the talos script actually exits with the appropriate self.return_code.

To do that we may have to add more self.buildbot_status() calls in mozharness.mozilla.testing.talos.  However, iirc, talos never goes orange, only green/red/retry.
Comment 23 Ed Morley [:emorley] 2013-07-19 11:53:29 PDT
(In reply to Aki Sasaki [:aki] from comment #22)
> However, iirc, talos never goes orange,
> only green/red/retry.

Talos also goes orange for crashes and test-unexpected-fails as of bug 829728, via:
https://hg.mozilla.org/build/buildbotcustom/file/f24d9219c221/steps/talos.py#l115
Comment 24 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-19 12:39:22 PDT
Moved the talos return codes to bug 896015.
Comment 25 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:44:29 PDT
Created attachment 779916 [details] [diff] [review]
talos_mozharness.diff

Enable talos mozharness on all FF25 branches.
Comment 26 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:45:17 PDT
Created attachment 779917 [details] [diff] [review]
different_builders.diff

This shows which builders are being modified
Comment 27 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:49:48 PDT
Created attachment 779924 [details] [diff] [review]
[mc] Match talos.zip to talos_revision
Comment 28 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:52:23 PDT
Created attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo
Comment 29 Joel Maher (:jmaher) 2013-07-23 11:53:24 PDT
Comment on attachment 779924 [details] [diff] [review]
[mc] Match talos.zip to talos_revision

Review of attachment 779924 [details] [diff] [review]:
-----------------------------------------------------------------

woot!
Comment 30 Joel Maher (:jmaher) 2013-07-23 11:54:18 PDT
Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779927 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/talos/talos.json
@@ +4,5 @@
>          "path": ""
>      },
>      "global": {
> +        "talos_repo": "http://hg.mozilla.org/build/talos",
> +        "talos_revision": "a11542b55a70"

the revision is wrong here
Comment 31 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:55:40 PDT
Created attachment 779932 [details] [diff] [review]
[mb] Match talos.zip to talos_revision and use talos_repo
Comment 32 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:55:55 PDT
Created attachment 779933 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo
Comment 33 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 11:57:05 PDT
Thanks for catching the mismatched revision.

Even though we're *not* enabling the new talos on older branches and will ride the trains (I think we can revisit this if we want to), I want talos.json to look the right way.
Comment 34 Joel Maher (:jmaher) 2013-07-23 12:01:46 PDT
Comment on attachment 779933 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779933 [details] [diff] [review]:
-----------------------------------------------------------------

this is not a valid revision :)
Comment 35 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 12:07:06 PDT
Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

This one seems correct:
http://hg.mozilla.org/build/talos/rev/a11542b55a70
Comment 36 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 12:08:25 PDT
Created attachment 779935 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo

I have no idea what that was about. Sorry!

http://hg.mozilla.org/build/talos/rev/560806cfa208
Comment 37 Joel Maher (:jmaher) 2013-07-23 12:10:13 PDT
Comment on attachment 779927 [details] [diff] [review]
[ma] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779927 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/talos/talos.json
@@ +4,5 @@
>          "path": ""
>      },
>      "global": {
> +        "talos_repo": "http://hg.mozilla.org/build/talos",
> +        "talos_revision": "a11542b55a70"

ok, this is the right revision for the right branch.
Comment 38 Joel Maher (:jmaher) 2013-07-23 12:10:37 PDT
Comment on attachment 779935 [details] [diff] [review]
[mr] Match talos.zip to talos_revision and use talos_repo

Review of attachment 779935 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!
Comment 39 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 12:18:06 PDT
I think we're going to have to postpone enabling this tomorrow since I would not like to start using an incorrect talos_revision and possibly cause regressions for using an older talos versions.
It's unfortunate but I think it is the safest.
I will have to keep an eye on newer talos.zip revision requests.

Anyone objects? I was thinking of pushing it to Monday.

On another note, can I land a patch on mozilla-central without waiting for a merge from m-i? or would I cause a merge conflict?

https://hg.mozilla.org/integration/mozilla-inbound/rev/0d4ab37e3f3e
https://hg.mozilla.org/releases/mozilla-aurora/rev/ac9b464cb3c8
https://hg.mozilla.org/releases/mozilla-beta/rev/64046aafb054
https://hg.mozilla.org/releases/mozilla-release/rev/481145f83cc6
Comment 40 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 12:50:22 PDT
Created attachment 779958 [details] [diff] [review]
update talos.json to match values from config.py

I'm waiting on jyeo to let me know if we're making use of "suites" inside of talos.json yet.

If we're, then we have to wait for this patch to spread across the branches.
Comment 41 Joel Maher (:jmaher) 2013-07-23 12:54:34 PDT
Comment on attachment 779958 [details] [diff] [review]
update talos.json to match values from config.py

Review of attachment 779958 [details] [diff] [review]:
-----------------------------------------------------------------

update buildbot-configs first, I backed out some stuff, sorry.

::: testing/talos/talos.json
@@ +63,5 @@
> +                "ignore_first:5",
> +                "--filter",
> +                "median"
> +            ]
> +        },

we can delete both of these: otherx, and svgx.

@@ +74,5 @@
>                  "median"
>              ]
>          },
> +        "rafx": {
> +            "tests": ["tcanvasmark"],

this should have tsvgx, tscrollx
Comment 42 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 13:14:38 PDT
Created attachment 779975 [details] [diff] [review]
update talos.json to match values from config.py
Comment 43 Joel Maher (:jmaher) 2013-07-23 13:18:04 PDT
Comment on attachment 779975 [details] [diff] [review]
update talos.json to match values from config.py

Review of attachment 779975 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great!
Comment 44 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 13:29:32 PDT
Comment on attachment 779975 [details] [diff] [review]
update talos.json to match values from config.py

I've landed it without removing the talos.zip block as that would have been bad:
https://hg.mozilla.org/integration/mozilla-inbound/rev/496a7582cf9e
Comment 45 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-23 13:30:29 PDT
Testing needed before we can enable talos mozharness across the board:
https://tbpl.mozilla.org/?tree=Try&rev=b5feca0c0c50
https://hg.mozilla.org/projects/cedar/rev/7cfd536a3d8a
Comment 46 John Hopkins (:jhopkins) 2013-07-23 13:43:04 PDT
Comment on attachment 779916 [details] [diff] [review]
talos_mozharness.diff

As discussed on IRC:
armenzg: try, cedar and ash should not change
jhopkins: "Rev4 MacOSX Lion 10.7 try talos" seems to have changed

I was comparing dump_master.py output before and after the patch and noticed the changed builder factories (note that 'try opt test' is unchanged but 'try talos' does indeed use a different factory than before):

< Rev4 MacOSX Lion 10.7 try opt test mochitest-browser-chrome ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test mochitest-browser-chrome ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test mochitest-other ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test mochitest-other ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test reftest-ipc ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test reftest-ipc ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test reftest ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test reftest ScriptFactory
< Rev4 MacOSX Lion 10.7 try opt test xpcshell ScriptFactory
> Rev4 MacOSX Lion 10.7 try opt test xpcshell ScriptFactory
< Rev4 MacOSX Lion 10.7 try talos chromez ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos chromez TalosFactory
< Rev4 MacOSX Lion 10.7 try talos dirtypaint ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos dirtypaint TalosFactory
< Rev4 MacOSX Lion 10.7 try talos dromaeojs ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos dromaeojs TalosFactory
< Rev4 MacOSX Lion 10.7 try talos other ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos other TalosFactory
< Rev4 MacOSX Lion 10.7 try talos svgr ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos svgr TalosFactory
< Rev4 MacOSX Lion 10.7 try talos tp5o ScriptFactory
> Rev4 MacOSX Lion 10.7 try talos tp5o TalosFactory
Comment 47 Ryan VanderMeulen [:RyanVM] 2013-07-23 17:46:58 PDT
https://hg.mozilla.org/mozilla-central/rev/0d4ab37e3f3e
Comment 48 Ed Morley [:emorley] 2013-07-24 05:44:39 PDT
https://hg.mozilla.org/mozilla-central/rev/496a7582cf9e
Comment 49 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-24 08:53:11 PDT
Talos.zip is needed for mobile. Re-testing:
https://tbpl.mozilla.org/?tree=Cedar

Status summary:
###############
* I need to provide a patch for Monday to enable talos mozharnes for all FF25 trees
* I want to make sure that the talos.json modifications from comment 44 do not cause any regressions on Monday
* I want to make sure that talos.json is consistent across older branches in case a developer wants to push a change to the try server (since it will be using talos mozharness).

[1] https://hg.mozilla.org/integration/mozilla-inbound/rev/496a7582cf9e
Comment 50 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-25 08:39:44 PDT
> * I want to make sure that talos.json is consistent across older branches in
> case a developer wants to push a change to the try server (since it will be
> using talos mozharness).

https://hg.mozilla.org/releases/mozilla-aurora/rev/729725cf05b8
https://hg.mozilla.org/releases/mozilla-beta/rev/d78022c9d69c
https://hg.mozilla.org/releases/mozilla-release/rev/2788eff35d00
https://hg.mozilla.org/releases/mozilla-b2g18/rev/7d759454371c
https://hg.mozilla.org/releases/mozilla-esr17/rev/c6d465f029f4
https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/c933e9b80897
Comment 51 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-25 09:27:45 PDT
Created attachment 781042 [details] [diff] [review]
enable talos mozharness for all FF25 trees

Only one line has been added.
Comment 52 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-25 11:22:12 PDT
I'm looking at this http://perf.snarkfest.net/compare-talos/index.html?oldRevs=f479167e92d2&newRev=7cfd536a3d8a&submit=true where I'm comparing two different revisions of Cedar I noticed a huge regression in tp5n_main_normal_fileio_paint but I can't find which job on Cedar runs it.
Any ideas?

https://tbpl.mozilla.org/?tree=Cedar&jobname=talos&showall=1
Comment 53 Joel Maher (:jmaher) 2013-07-25 11:52:19 PDT
I am not sure how to use the compare-talos toolchain, but we could look at some revisions on cedar and see what the raw values are.  This looks to be related to xperf, and those values are subject to change a lot.
Comment 54 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-25 13:19:59 PDT
This is tedious.
It would be great that compare-talos allowed to create graphs URLs to compare to other branches or within the same branch (that is what I do in the following links).

From:
http://perf.snarkfest.net/compare-talos/index.html?oldRevs=f479167e92d2&newRev=7cfd536a3d8a&submit=true

I've built the following URLs:
http://graphs.mozilla.org/graph.html#tests=[[224,63,24],[224,26,24]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[251,26,25],[251,63,25]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[244,26,25],[244,63,25]]&sel=none&displayrange=7&datatype=running
http://graphs.mozilla.org/graph.html#tests=[[245,26,25],[245,63,25]]&sel=none&displayrange=7&datatype=running
I would say that we're fine and it is due to noisy talos jobs.

Unless someone strongly believes that I should keep up posting URL for each regressions mentioned on compare-talos for more than 3% deviance I would like to assume that we're fine.

I'm facing the same situation as to went I tried if talos-mozharness causes clear regressions compared to buildbotcustom/factory.py. It requires lots of eye analysis.

Missing URLs for these regressions:
- Ts Paint, MED Dirty Profile (tspaint_places_generated_med) - Mac 10.8 
- Ts Paint, MAX Dirty Profile (tspaint_places_generated_max) - Mac 10.8 & Ub. 32 
- TResize (tresize) - Win8
- Tp5 Optimized Responsiveness (tp5o_responsiveness_paint) - Mac 10.8, Win7 & Win8
- Tp5 Optimized MozAfterPaint (tp5o_shutdown_paint) - all but win7 & win8
- Tp5 Optimized (Modified Page List Bytes) (tp5o_modlistbytes_paint) - Win7
Comment 55 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-29 08:35:18 PDT
We'll deploy this tomorrow morning.
Review is expected to come through today.
Comment 56 John Hopkins (:jhopkins) 2013-07-29 12:03:06 PDT
Comment on attachment 781042 [details] [diff] [review]
enable talos mozharness for all FF25 trees

armenzg: can you please post an updated patch?  This one no longer applies cleanly to buildbot-configs.
Comment 57 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-29 12:12:14 PDT
Created attachment 782698 [details] [diff] [review]
enable talos mozharness for all FF25 trees (updated)
Comment 58 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-29 12:51:11 PDT
Created attachment 782716 [details]
Results from firt changeset when we enabled talos mozharness on Cedar: 14e3e9ab9994
Comment 59 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-29 12:52:19 PDT
Created attachment 782718 [details]
Results from talos.json updates on Cedar: 7cfd536a3d8a
Comment 60 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-29 14:25:04 PDT
I don't really have a clear way to determine what suites will clearly be affected by the talos mozharness tomorrow.

The last two attachments were obtained with this:
python talos/compare.py --revision=14e3e9ab9994 --branch=Cedar --masterbranch=Firefox --print-graph-url
python talos/compare.py --revision=14e3e9ab9994 --branch=Cedar --masterbranch=Firefox --print-graph-url

I wanted to understand things a little better and be sure that I know exactly what is going to change so I can answer any questions from developers. I wanted to be more educated in a way.

For now, I will re-paste the analysis that jmaher did a bit ago and hope that nothing new has cropped in.

From bug 802801:

(In reply to Joel Maher (:jmaher) from comment #10)
> So it appears that our ts test (start/stop the browser 20 times) is
> problematic.  the places_med|max are just different profiles used while
> running the test.  I have verified this on a few different changesets.  
> 
> I really don't understand how changing to mozharness could cause this, but
> maybe there is some additional overhead induced on the system with
> mozharness when it comes to launching a new process.  
> 
> We could take this as a bump in the numbers and accept that.  I am open to
> any thoughts here.
Comment 61 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-30 07:14:35 PDT
This is live now.
Comment 62 Armen Zambrano [:armenzg] - Engineering productivity 2013-07-30 08:35:42 PDT
This is probably one of the expected regressions to be reported:
http://graphs.mozilla.org/graph.html#tests=[[226,94,25],[227,94,25],[226,131,25],[227,131,25]]&sel=1375122199529,1375198347356&displayrange=7&datatype=running
Comment 63 Aki Sasaki [:aki] 2013-10-16 11:39:56 PDT
I'm going to guess we're done here... ?
I don't think the two existing blocker bugs are actually strictly blocking.
Comment 64 Jonathan Griffin (:jgriffin) 2013-10-16 13:12:00 PDT
I agree; I think this can be closed.

Note You need to log in before you can comment on or make changes to this bug.