1220296 - Restart buildbot masters more frequently

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris AtLee [:catlee]]

Attachment: master-age.diff

We have nagios alerts set up to restart the masters after 41 days. I think we need to cut this in half - to 20 days.

Comment 1

[Amy Rich [:arr] [:arich]]

Comment on attachment 8681493 [details] [diff] [review]
master-age.diff

Review of attachment 8681493 [details] [diff] [review]:
-----------------------------------------------------------------

FYI, we just got done increasing this number because people only wanted to reboot during a TCW every 6+ weeks. We need an accompanying process change to go along with the nagios change.

Comment 2

[Chris AtLee [:catlee]]

That's true...but I don't think there's a particular need to do this inside a TCW. Slow rolling restarts should be ok.

Comment 3

[Hal Wine [:hwine] (leaving Jan 17, ping on slack if needed)]

any timing adjustments should be reflected in bug 1197853 - fwiw, we cancelled October's restart based on report that is wasn't needed (and would have been on TCW activity)

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I think I did the October restart anyway as nagios was alerting. 

+1 to restarting more frequently. I think we need to take assorted tools we have and productionise them. Pretty sure coop has something, and I've kept catlee's fabric enhancements going at https://github.com/nthomas-mozilla/build-tools/tree/fabric (this got flaky on the last few masters when I used it, for unknown reasons).

Comment 5

[Armen [:armenzg]]

Do we have a check to detect the masters starting to get into this bad state?

I would assume it won't be hard to sell rebooting the masters more often if it help us keeping the Windows throughput better. Even if there is an increased risk for the master rebooting and getting into a bad state for a short time.

Comment 6

[Chris Cooper [:coop] (he/him)]

My script is here: https://github.com/ccooper/build-tools/blob/master/buildfarm/maintenance/restart_masters.py

I ran it this weekend to restart all the masters. It's not perfect -- we hung on two masters requiring manual intervention -- but we could certainly dig into those issues and fix them. We could schedule the script to trigger restarts every weekend without much issue.