661523 - automatically gracefully restart buildbot masters every week

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task, P5)

Description

[Chris AtLee [:catlee]]

The idea here is to avoid long-running processes which suffer from things like memory leaks. It'd be great to fix those, but we don't always have time to.

I'm thinking something like this...On Sunday night, for each pool of masters, gracefully restart each master in a pool in series. Pools can be processed in parallel.

On Monday morning, buildduty (with help from nagios!) checks to see if the restarts completed successfully, or if they're hung somewhere.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This sounds pretty difficult.  Let's get periodic *manual* restarts going first.  I'd suggest doing this with slavealloc - bring a new instance up, disable the old in slavealloc, and graceful it.  Then we don't lose capacity at any point.

Anyway, futuristic for now..

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage.

Comment 3

[[e-mail] (:bhearsum)]

Do we still care about this now that we're moving most things to Taskcluster?

Comment 4

[Chris AtLee [:catlee]]

We have nagios alerts for buildbot process age, and we restart them manually periodically.