The default bug view has changed. See this FAQ.

Get catlee's cool fabric actions in production for e.g. managing pulse queues on buildbot masters

RESOLVED FIXED

Status

Release Engineering
Tools
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: pmoore, Assigned: pmoore)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Assignee)

Description

3 years ago
Created attachment 8426236 [details] [diff] [review]
catlee-fabric-actions.patch

Catlee has written some extra fabric actions for managing buildbot masters, such as manipulating pulse queues.

https://github.com/catlee/tools/compare/master...fabric#diff-8fd5c265985d88a725d36d2e7fddc64aR303

Let's get these reviewed and in production. :)
Attachment #8426236 - Flags: review?(bugspam.Callek)
Attachment #8426236 - Flags: feedback?(catlee)
(Assignee)

Comment 1

3 years ago
Catlee says there are new dependencies introduced by the patch, which need addressing before it is reviewed. I'll remove the review flags...
(Assignee)

Updated

3 years ago
Attachment #8426236 - Flags: review?(bugspam.Callek)
Attachment #8426236 - Flags: feedback?(catlee)

Updated

3 years ago
Blocks: 1013511
(Assignee)

Comment 2

3 years ago
Created attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

This is a subset of the actions that nthomas kindly gave me, which I was able to use to reset the queues on 1 Sep 2014 with bug 1061208.

Not obsoleting the other patch, in case there are other parts we still also want to grab.
Attachment #8484322 - Flags: review?(nthomas)
Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

action_oldest_twistdlog and action_master_health are actually my hackery, but r+ on landing action_retry_dead_queue.
Attachment #8484322 - Flags: review?(nthomas) → review+
Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

Review of attachment 8484322 [details] [diff] [review]:
-----------------------------------------------------------------

since nick disavows ability to r+ the others, allow me to do so.
Attachment #8484322 - Flags: review+
action_oldest_twistdlog is really just cruft from a one-off weird case.
(Assignee)

Comment 6

3 years ago
OK, I'll land action_master_health and action_retry_dead_queue...
(Assignee)

Comment 7

3 years ago
Comment on attachment 8484322 [details] [diff] [review]
bug1013961_tools_v1.patch

Checked in, with the exception of the action_oldest_twistdlog method...

I'll leave this bug open for now, as there may be other methods we want to mine from the original attachment.
Attachment #8484322 - Flags: checked-in+
(Assignee)

Comment 8

3 years ago
Created attachment 8496049 [details] [diff] [review]
bug1013961_tools_uptime_v1.patch

Cheekily tagging this onto this bug here, as an extra action I've added for seeing uptime of buildbot masters...
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #8496049 - Flags: review?(coop)
(Assignee)

Comment 9

3 years ago
Output looks like this:
(fabric)pmoore@Elisandra:~/git/tools master $ manage_masters.py -f buildfarm/maintenance/production-masters.json -R all -j 32 uptime
bm85-build1               09:29:15 up 324 days, 18:00,  1 user,  load average: 0.08, 0.05, 0.02
bm82-build1               09:29:15 up 324 days, 18:02,  1 user,  load average: 0.03, 0.05, 0.06
bm81-build_scheduler      09:29:15 up 436 days,  2:37,  2 users,  load average: 0.22, 0.23, 0.25
bm81-tests_scheduler      09:29:15 up 436 days,  2:37,  1 user,  load average: 0.22, 0.23, 0.25
bm84-build1               09:29:15 up 324 days, 18:00,  1 user,  load average: 0.20, 0.08, 0.04
bm83-try1                 09:29:15 up 324 days, 18:02,  1 user,  load average: 0.20, 0.17, 0.07
bm05-tests1-linux32       09:29:15 up  2:25,  1 user,  load average: 0.04, 0.13, 0.07
bm06-tests1-linux32       09:29:15 up  1:36,  1 user,  load average: 0.00, 0.02, 0.04
bm04-tests1-linux32       09:29:15 up  1:40,  1 user,  load average: 0.00, 0.03, 0.08
bm53-tests1-linux64       09:29:15 up 547 days, 37 min,  1 user,  load average: 1.14, 1.24, 1.23
bm68-tests1-linux64       09:29:15 up 534 days, 11 min,  1 user,  load average: 1.22, 1.18, 1.14
bm54-tests1-linux64       09:29:15 up 546 days, 23:50,  1 user,  load average: 1.07, 1.16, 1.16
bm72-build1               09:29:15 up 22:39,  2 users,  load average: 0.04, 0.06, 0.11
bm73-build1               09:29:15 up 204 days,  6:00,  1 user,  load average: 0.28, 0.17, 0.12
bm79-try1                 09:29:15 up 203 days,  2:11,  1 user,  load average: 0.16, 0.07, 0.05
bm74-build1               09:29:15 up 204 days,  6:17,  1 user,  load average: 0.02, 0.24, 0.25
bm78-try1                 09:29:15 up 203 days,  3:20,  1 user,  load average: 0.00, 0.07, 0.05
bm86-build1               09:29:16 up 324 days, 18:00,  1 user,  load average: 0.08, 0.10, 0.09
bm87-try1                 09:29:16 up 324 days, 18:00,  1 user,  load average: 0.08, 0.05, 0.01
bm89-tests1-panda         09:29:16 up 282 days, 23:51,  1 user,  load average: 0.19, 0.14, 0.15
bm52-tests1-linux64       09:29:16 up 223 days, 23:12,  1 user,  load average: 1.06, 1.23, 1.11
bm51-tests1-linux64       09:29:16 up 547 days,  1:32,  1 user,  load average: 0.97, 1.06, 0.95
bm71-build1               09:29:16 up 204 days,  7:04,  1 user,  load average: 1.20, 0.57, 0.36
bm67-tests1-linux64       09:29:16 up 534 days, 44 min,  1 user,  load average: 0.71, 0.78, 0.87
bm77-build1               09:29:16 up 203 days,  4:21,  1 user,  load average: 0.42, 0.47, 0.35
bm76-try1                 09:29:16 up 203 days,  4:38,  1 user,  load average: 0.07, 0.12, 0.19
bm01-tests1-linux32       09:29:16 up 23:16,  1 user,  load average: 1.68, 1.97, 1.92
bm75-try1                 09:29:16 up 191 days, 22:59,  1 user,  load average: 0.01, 0.12, 0.18
bm03-tests1-linux32       09:29:16 up 191 days, 22:03,  1 user,  load average: 1.91, 1.82, 1.76
bm02-tests1-linux32       09:29:16 up 191 days, 22:04,  1 user,  load average: 1.87, 1.73, 1.66
bm70-build1               09:29:16 up 207 days, 18:21,  1 user,  load average: 1.06, 0.37, 0.36
bm91-build1               09:29:16 up  3:03,  1 user,  load average: 0.08, 0.11, 0.14
bm100-tests1-panda        09:29:17 up 280 days, 23:36,  1 user,  load average: 0.23, 0.14, 0.14
bm102-tests1-panda        09:29:17 up 280 days, 23:41,  1 user,  load average: 0.13, 0.13, 0.15
bm101-tests1-panda        09:29:17 up 281 days,  1:21,  1 user,  load average: 0.05, 0.09, 0.12
bm104-tests1-linux        09:29:17 up 224 days, 22:38,  1 user,  load average: 0.12, 0.12, 0.09
bm103-tests1-linux        09:29:17 up 224 days, 21:46,  1 user,  load average: 0.00, 0.00, 0.00
bm105-tests1-linux        09:29:17 up 224 days, 22:38,  1 user,  load average: 0.00, 0.00, 0.00
bm106-tests1-macosx       09:29:17 up 224 days, 22:37,  1 user,  load average: 0.26, 0.57, 0.64
bm108-tests1-macosx       09:29:17 up 224 days, 22:36,  1 user,  load average: 0.62, 0.79, 0.75
bm107-tests1-macosx       09:29:17 up 224 days, 22:37,  1 user,  load average: 1.47, 0.99, 0.78
bm109-tests1-windows      09:29:18 up 224 days, 22:35,  1 user,  load average: 1.51, 1.22, 1.11
bm110-tests1-windows      09:29:18 up 224 days, 22:35,  1 user,  load average: 1.02, 1.10, 1.12
bm112-tests1-windows      09:29:18 up 224 days, 22:34,  1 user,  load average: 1.57, 1.41, 1.22
bm111-tests1-windows      09:29:18 up 224 days, 22:29,  1 user,  load average: 1.11, 1.13, 1.14
bm115-tests1-linux64      09:29:18 up 63 days, 14:02,  1 user,  load average: 0.68, 0.76, 0.91
bm94-build1               09:29:18 up 191 days, 17:32,  1 user,  load average: 0.09, 0.13, 0.28
bm116-tests1-linux64      09:29:19 up 60 days, 17:20,  1 user,  load average: 0.62, 0.85, 1.02
bm118-tests1-linux64      09:29:19 up  3:23,  1 user,  load average: 0.59, 0.67, 0.88
bm114-tests1-linux64      09:29:19 up 60 days, 17:12,  1 user,  load average: 0.67, 0.79, 0.81
bm113-tests1-linux64      09:29:19 up 60 days, 17:20,  1 user,  load average: 0.85, 0.67, 0.69
bm117-tests1-linux64      09:29:19 up 66 days, 19:26,  1 user,  load average: 0.98, 0.93, 0.79
================================================================================
(fabric)pmoore@Elisandra:~/git/tools master $

Updated

3 years ago
Attachment #8496049 - Flags: review?(coop) → review+
(Assignee)

Comment 10

3 years ago
Comment on attachment 8496049 [details] [diff] [review]
bug1013961_tools_uptime_v1.patch

http://hg.mozilla.org/build/tools/rev/2d2c52fdfc4b
Attachment #8496049 - Flags: checked-in+

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.