1013961 - Get catlee's cool fabric actions in production for e.g. managing pulse queues on buildbot masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

Attachment: catlee-fabric-actions.patch

Catlee has written some extra fabric actions for managing buildbot masters, such as manipulating pulse queues.

https://github.com/catlee/tools/compare/master...fabric#diff-8fd5c265985d88a725d36d2e7fddc64aR303

Let's get these reviewed and in production. :)

Comment 1

[Pete Moore [:pmoore][:pete]]

Catlee says there are new dependencies introduced by the patch, which need addressing before it is reviewed. I'll remove the review flags...

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: bug1013961_tools_v1.patch

This is a subset of the actions that nthomas kindly gave me, which I was able to use to reset the queues on 1 Sep 2014 with bug 1061208.

Not obsoleting the other patch, in case there are other parts we still also want to grab.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

action_oldest_twistdlog and action_master_health are actually my hackery, but r+ on landing action_retry_dead_queue.

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

Review of attachment 8484322 [details] [diff] [review]:
-----------------------------------------------------------------

since nick disavows ability to r+ the others, allow me to do so.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

action_oldest_twistdlog is really just cruft from a one-off weird case.

Comment 6

[Pete Moore [:pmoore][:pete]]

OK, I'll land action_master_health and action_retry_dead_queue...

Comment 7

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

Checked in, with the exception of the action_oldest_twistdlog method...

I'll leave this bug open for now, as there may be other methods we want to mine from the original attachment.

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: bug1013961_tools_uptime_v1.patch

Cheekily tagging this onto this bug here, as an extra action I've added for seeing uptime of buildbot masters...

Comment 9

[Pete Moore [:pmoore][:pete]]

Output looks like this:
(fabric)pmoore@Elisandra:~/git/tools master $ manage_masters.py -f buildfarm/maintenance/production-masters.json -R all -j 32 uptime
bm85-build1               09:29:15 up 324 days, 18:00,  1 user,  load average: 0.08, 0.05, 0.02
bm82-build1               09:29:15 up 324 days, 18:02,  1 user,  load average: 0.03, 0.05, 0.06
bm81-build_scheduler      09:29:15 up 436 days,  2:37,  2 users,  load average: 0.22, 0.23, 0.25
bm81-tests_scheduler      09:29:15 up 436 days,  2:37,  1 user,  load average: 0.22, 0.23, 0.25
bm84-build1               09:29:15 up 324 days, 18:00,  1 user,  load average: 0.20, 0.08, 0.04
bm83-try1                 09:29:15 up 324 days, 18:02,  1 user,  load average: 0.20, 0.17, 0.07
bm05-tests1-linux32       09:29:15 up  2:25,  1 user,  load average: 0.04, 0.13, 0.07
bm06-tests1-linux32       09:29:15 up  1:36,  1 user,  load average: 0.00, 0.02, 0.04
bm04-tests1-linux32       09:29:15 up  1:40,  1 user,  load average: 0.00, 0.03, 0.08
bm53-tests1-linux64       09:29:15 up 547 days, 37 min,  1 user,  load average: 1.14, 1.24, 1.23
bm68-tests1-linux64       09:29:15 up 534 days, 11 min,  1 user,  load average: 1.22, 1.18, 1.14
bm54-tests1-linux64       09:29:15 up 546 days, 23:50,  1 user,  load average: 1.07, 1.16, 1.16
bm72-build1               09:29:15 up 22:39,  2 users,  load average: 0.04, 0.06, 0.11
bm73-build1               09:29:15 up 204 days,  6:00,  1 user,  load average: 0.28, 0.17, 0.12
bm79-try1                 09:29:15 up 203 days,  2:11,  1 user,  load average: 0.16, 0.07, 0.05
bm74-build1               09:29:15 up 204 days,  6:17,  1 user,  load average: 0.02, 0.24, 0.25
bm78-try1                 09:29:15 up 203 days,  3:20,  1 user,  load average: 0.00, 0.07, 0.05
bm86-build1               09:29:16 up 324 days, 18:00,  1 user,  load average: 0.08, 0.10, 0.09
bm87-try1                 09:29:16 up 324 days, 18:00,  1 user,  load average: 0.08, 0.05, 0.01
bm89-tests1-panda         09:29:16 up 282 days, 23:51,  1 user,  load average: 0.19, 0.14, 0.15
bm52-tests1-linux64       09:29:16 up 223 days, 23:12,  1 user,  load average: 1.06, 1.23, 1.11
bm51-tests1-linux64       09:29:16 up 547 days,  1:32,  1 user,  load average: 0.97, 1.06, 0.95
bm71-build1               09:29:16 up 204 days,  7:04,  1 user,  load average: 1.20, 0.57, 0.36
bm67-tests1-linux64       09:29:16 up 534 days, 44 min,  1 user,  load average: 0.71, 0.78, 0.87
bm77-build1               09:29:16 up 203 days,  4:21,  1 user,  load average: 0.42, 0.47, 0.35
bm76-try1                 09:29:16 up 203 days,  4:38,  1 user,  load average: 0.07, 0.12, 0.19
bm01-tests1-linux32       09:29:16 up 23:16,  1 user,  load average: 1.68, 1.97, 1.92
bm75-try1                 09:29:16 up 191 days, 22:59,  1 user,  load average: 0.01, 0.12, 0.18
bm03-tests1-linux32       09:29:16 up 191 days, 22:03,  1 user,  load average: 1.91, 1.82, 1.76
bm02-tests1-linux32       09:29:16 up 191 days, 22:04,  1 user,  load average: 1.87, 1.73, 1.66
bm70-build1               09:29:16 up 207 days, 18:21,  1 user,  load average: 1.06, 0.37, 0.36
bm91-build1               09:29:16 up  3:03,  1 user,  load average: 0.08, 0.11, 0.14
bm100-tests1-panda        09:29:17 up 280 days, 23:36,  1 user,  load average: 0.23, 0.14, 0.14
bm102-tests1-panda        09:29:17 up 280 days, 23:41,  1 user,  load average: 0.13, 0.13, 0.15
bm101-tests1-panda        09:29:17 up 281 days,  1:21,  1 user,  load average: 0.05, 0.09, 0.12
bm104-tests1-linux        09:29:17 up 224 days, 22:38,  1 user,  load average: 0.12, 0.12, 0.09
bm103-tests1-linux        09:29:17 up 224 days, 21:46,  1 user,  load average: 0.00, 0.00, 0.00
bm105-tests1-linux        09:29:17 up 224 days, 22:38,  1 user,  load average: 0.00, 0.00, 0.00
bm106-tests1-macosx       09:29:17 up 224 days, 22:37,  1 user,  load average: 0.26, 0.57, 0.64
bm108-tests1-macosx       09:29:17 up 224 days, 22:36,  1 user,  load average: 0.62, 0.79, 0.75
bm107-tests1-macosx       09:29:17 up 224 days, 22:37,  1 user,  load average: 1.47, 0.99, 0.78
bm109-tests1-windows      09:29:18 up 224 days, 22:35,  1 user,  load average: 1.51, 1.22, 1.11
bm110-tests1-windows      09:29:18 up 224 days, 22:35,  1 user,  load average: 1.02, 1.10, 1.12
bm112-tests1-windows      09:29:18 up 224 days, 22:34,  1 user,  load average: 1.57, 1.41, 1.22
bm111-tests1-windows      09:29:18 up 224 days, 22:29,  1 user,  load average: 1.11, 1.13, 1.14
bm115-tests1-linux64      09:29:18 up 63 days, 14:02,  1 user,  load average: 0.68, 0.76, 0.91
bm94-build1               09:29:18 up 191 days, 17:32,  1 user,  load average: 0.09, 0.13, 0.28
bm116-tests1-linux64      09:29:19 up 60 days, 17:20,  1 user,  load average: 0.62, 0.85, 1.02
bm118-tests1-linux64      09:29:19 up  3:23,  1 user,  load average: 0.59, 0.67, 0.88
bm114-tests1-linux64      09:29:19 up 60 days, 17:12,  1 user,  load average: 0.67, 0.79, 0.81
bm113-tests1-linux64      09:29:19 up 60 days, 17:20,  1 user,  load average: 0.85, 0.67, 0.69
bm117-tests1-linux64      09:29:19 up 66 days, 19:26,  1 user,  load average: 0.98, 0.93, 0.79
================================================================================
(fabric)pmoore@Elisandra:~/git/tools master $

Comment 10

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8496049 [details] [diff] [review]
bug1013961_tools_uptime_v1.patch

http://hg.mozilla.org/build/tools/rev/2d2c52fdfc4b