875822 - buildslaves on foopies should exit after each run.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

With the addition of buildbot masters now supporting "Slave initiated shutdown" (e.g. graceful) in a way that does not require us to curl the buildbot WebStatus page, we should make use of this.

For the "Why should we do this":
* Allows us to stop killing the buildslave when we see an error.flg
* Allows us to use slavealloc for enabled/disabled/what-master state for mobile devices.
* Allows us to run verify.py *before* the job, eliminating many developer/sheriff-viewable issues.
* Allows us to improve end-to-end time by reducing the run-by-buildbot verify.py checks.

This is meant to be a pretty standalone improvement whose only improvement is side-effect of running verify.py outside of buildbot after every job, before the next buildbot run starts.

Comment 1

[Justin Wood (:Callek)]

Attachment: [device buildbot.tac] manual diff

To be manually applied to our hundreds of devices. Yea sadface.

(r? to catlee since he wrote this upstream patch and backported it to our end to begin with)

Comment 2

[Justin Wood (:Callek)]

Attachment: [buildbotcustom] graceful the slave

This actual buildbot patch is untested, but the concept of touching a shutdown.stamp after a job has started shutting down the job if buildbot was started with the change to buildbot.tac has been.

I will test this in staging before deploying this one.

Comment 3

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #2)
> Created attachment 753858 [details] [diff] [review]
> [buildbotcustom] graceful the slave
> 
> I will test this in staging before deploying this one.

Tested and looks good, now just awaiting catlee's review.

Comment 4

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/buildbotcustom/rev/ac664092d9d8

Comment 5

[Armen [:armenzg]]

Live in production.

Comment 7

[Justin Wood (:Callek)]

This is now live, and seems to be holding steady with no ill effects.