795019 - check clobberer.py and count_and_reboot.py into mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Aki Sasaki (not active)]

We've got hgtool.py; these two should round out what we need from build/tools on a consistent basis.

Comment 1

[Aki Sasaki (not active)]

Attachment: mozharness/external_tools/

Should allow us to stop cloning build/tools for these.
ScriptFactory can point at count_and_reboot.py, but we'll need something in mozharness (BuildbotMixin?) to call clobberer.py.

Comment 2

[Aki Sasaki (not active)]

These are the same as the build/tools versions, except chmod'ed a+x, and I changed one from #!/usr/bin/python to #!/usr/bin/env python.

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 665540 [details] [diff] [review]
mozharness/external_tools/

Review of attachment 665540 [details] [diff] [review]:
-----------------------------------------------------------------

should purge_builds.py be added too?

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 665540 [details] [diff] [review]
mozharness/external_tools/

Checked in, with purge_builds.py.

Still need mozharness patches to use clobberer.py and purge_builds.py, and buildbot-configs patches to change the location of count_and_reboot.py for mozharness scripts.

We can probably also remove build/tools from the repo list for a number of jobs now.

Comment 5

[Aki Sasaki (not active)]

( http://hg.mozilla.org/build/mozharness/rev/4d9828f58cb4 )

Comment 6

[Aki Sasaki (not active)]

Attachment: call count_and_reboot.py from external_tools/

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 666047 [details] [diff] [review]
call count_and_reboot.py from external_tools/

IRC review from Catlee.
http://hg.mozilla.org/build/buildbot-configs/rev/9c0386f3e28d

Comment 8

[Aki Sasaki (not active)]

Attachment: stop pulling hg.m.o/build/tools for count_and_reboot.py

Since we have a copy of count_and_reboot.py in mozharness/external_tools/ now, we can remove the requirement of cloning hg.m.o/build/tools for that file.

Updated marionette, peptest, talos, unittests to no longer need to pull that repo.  Since that was the only repo we were pulling, I removed that action and the repos config entries.

Jetperf still needs to clone addon sdk repositories, so I left it alone.

A few new talos docstrings crept their way into my testing repo -- I can remove them from the review or leave them.

Comment 9

[Aki Sasaki (not active)]

We're now using the in-mozharness version of count_and_reboot.py in production.

Comment 10

[Jonathan Griffin (:jgriffin)]

Comment on attachment 666063 [details] [diff] [review]
stop pulling hg.m.o/build/tools for count_and_reboot.py

Review of attachment 666063 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks!

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 666063 [details] [diff] [review]
stop pulling hg.m.o/build/tools for count_and_reboot.py

http://hg.mozilla.org/build/mozharness/rev/8e36db25e15f

Comment 12

[Aki Sasaki (not active)]

My current plan of attack for purge_builds.py and clobberer.py:

* create a clobber() method in BuildbotMixin.  I think this can override the BaseScript.clobber(), but I'd have to check

* BuildbotMixin.clobber() would check a config flag on whether to clobber full, partial, check with clobberer_url, or not at all.  If full (or clobberer_url says yes), BaseScript.clobber().  If partial, nuke a list of files/directories specified in the config.  If not at all, skip this portion.

We may have to default to partial or not at all for depend builds/tests.

* Then it would check for an option to purge builds.  This should be off by default for standalone/devs, and on by default for production.

I'm not entirely sure if I will be able to do this before PTO, but it's on my list of things to try to get done.

Comment 13

[Aki Sasaki (not active)]

[16:36]	<catlee-away>	aki: when are you PTO? I could tackle purge/clobber for mozharness
[16:36]	<catlee-away>	since we clearly need that for b2g builds

Right now it looks like I might not get to this, and this was causing b2g jobs to fill up our aws boxen.  Reassigning; thanks Catlee!