Closed Bug 1279394 Opened 10 years ago Closed 10 years ago

Disable half of the 10.6 pool

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: coop, Assigned: aobreja)

References

Details

Attachments

(1 file, 1 obsolete file)

Snow Leopard (10.6) is no longer supported on mozilla-central. 10.6 jobs are also no longer run by default on Try, so the volume of 10.6 testing is greatly reduced. We should be able to disable (at least) half of the 10.6 machine pool and still keep up with load. Order of operations: * Disable half the pool (68 machines) * Verify that we have no wait times on 10.6 * Assess whether we can disable more machines. Maybe we can get away with two thirds? * Remove all disabled 10.6 machines from buildbot-configs to simpilfy configs and speed up test masters by some fraction.
Blocks: 1279393
Assignee: nobody → andrei.obreja
Disabled 68 slaves (snow Leopard-10.6). mysql> select count(*) from slaves where enabled=0 and notes='Disabled in bug 1279394'; +----------+ | count(*) | +----------+ | 68 | +----------+ 1 row in set (0.01 sec) Now we are monitoring to see if the remain enabled slaves(51) can keep up with the load.
Disabled 21 more slaves (snow Leopard-10.6)as there were no issue for the remaining slaves to keep up with the load. mysql> select count(*) from slaves where enabled=0 and notes='Disabled in bug 1279394'; +----------+ | count(*) | +----------+ | 89 | +----------+ 1 row in set (0.00 sec) Now we are monitoring to see if the remain enabled slaves (30) are enough.
Attached patch bug1279394_bb_1.patch (obsolete) — — Splinter Review
Added patch to remove all disabled 10.6 machines from buildbot-configs.
Attachment #8762509 - Flags: review?(kmoir)
Attachment #8762509 - Flags: review?(coop)
Only 30 slave machines for 10.6 are still enabled and seems to keep up with the load: mysql> select count(*) from slaves where enabled=1 and name like 't-snow-r4%'; +----------+ | count(*) | +----------+ | 30 | +----------+ 1 row in set (0.00 sec)
Keep in mind that your testing so far has been over the weekend before a workweek, so all you've kept up with were blocklist updates that only landed on two of the three branches running 10.6, and then a single push. More reasonable load to plan for is a couple of pushes to each of the three branches, with relman sitting watching and waiting for results. And that you now absolutely unquestionably can't keep up with, because 30 slaves is exactly how many you need to run a set of tests on an opt build, and one short of how many you need to run a set of tests on a debug build. With 31 slaves, you could keep up with a single push to only one of the three branches at a time, only if the debug build finishes far enough ahead of the opt build to be done with tests before the opt build schedules its tests (and by looking at a couple of pushes, it looks like typically it doesn't, and there's a small bit of overlap), and only if nobody ever gets a 10.6 loaner and none of them ever break. But in your worst case, when someone pushes to try with 10.6 tests at just the wrong time right before a sheriff pushes a chemspill patch simultaneously to beta and release and esr45, you're looking at something close to 7 hours.
And a third of the still-enabled ones were already broken, or at least bent, having rotted away during the period of idleness since the load dropped. I think that's atypical and you don't need to plan on a third over the desired capacity, but I could well be wrong, not having seen how they behave around idleness other than over weekends.
Enabled back 21 slaves,now there are 51 enabled slaves: mysql> select count(*) from slaves where name like 't-snow-r4%' and enabled=1; +----------+ | count(*) | +----------+ | 51 | +----------+ 1 row in set (0.00 sec) Thank you Phil for the warning,I will keep this number of slaves until some test will be performed and then we will have more informations about how many slaves can keep up with the load.
Comment on attachment 8762509 [details] [diff] [review] bug1279394_bb_1.patch Review of attachment 8762509 [details] [diff] [review]: ----------------------------------------------------------------- ::: mozilla-tests/production_config.py @@ +37,4 @@ > for i in range(102, 103): # Use win8's 102 for win10 // Bug 1191481 > SLAVES['win10']['t-w864-ix-%03i' % i] = {} > > +for i range(99, 115) + range(116, 129): # slaves 0034&0093&0094&0153 have been decommed ; bug 1279394 - 89 slaves have been decomed The new range is fine, but you're missing the "in" in "for i in range()". r+ with that fixed.
Attachment #8762509 - Flags: review?(kmoir)
Attachment #8762509 - Flags: review?(coop)
Attachment #8762509 - Flags: review+
A new patch was added,with a new range as more tests need to be run to see if the enabled slaves can keep up with the load therefore some of the slaves were reenabled. Thank you Coop for review.
Attachment #8762509 - Attachment is obsolete: true
Attachment #8762663 - Flags: review?(coop)
Attachment #8762663 - Flags: review?(coop) → review+
Comment on attachment 8762663 [details] [diff] [review] bug1279394_bb_1.patch and merged to production
Attachment #8762663 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: