Closed
Bug 1279394
Opened 10 years ago
Closed 10 years ago
Disable half of the 10.6 pool
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Infrastructure & Operations Graveyard
CIDuty
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: coop, Assigned: aobreja)
References
Details
Attachments
(1 file, 1 obsolete file)
|
738 bytes,
patch
|
coop
:
review+
kmoir
:
checked-in+
|
Details | Diff | Splinter Review |
Snow Leopard (10.6) is no longer supported on mozilla-central. 10.6 jobs are also no longer run by default on Try, so the volume of 10.6 testing is greatly reduced. We should be able to disable (at least) half of the 10.6 machine pool and still keep up with load.
Order of operations:
* Disable half the pool (68 machines)
* Verify that we have no wait times on 10.6
* Assess whether we can disable more machines. Maybe we can get away with two thirds?
* Remove all disabled 10.6 machines from buildbot-configs to simpilfy configs and speed up test masters by some fraction.
| Assignee | ||
Updated•10 years ago
|
Assignee: nobody → andrei.obreja
| Assignee | ||
Comment 1•10 years ago
|
||
Disabled 68 slaves (snow Leopard-10.6).
mysql> select count(*) from slaves where enabled=0 and notes='Disabled in bug 1279394';
+----------+
| count(*) |
+----------+
| 68 |
+----------+
1 row in set (0.01 sec)
Now we are monitoring to see if the remain enabled slaves(51) can keep up with the load.
| Assignee | ||
Comment 2•10 years ago
|
||
Disabled 21 more slaves (snow Leopard-10.6)as there were no issue for the remaining slaves to keep up with the load.
mysql> select count(*) from slaves where enabled=0 and notes='Disabled in bug 1279394';
+----------+
| count(*) |
+----------+
| 89 |
+----------+
1 row in set (0.00 sec)
Now we are monitoring to see if the remain enabled slaves (30) are enough.
| Assignee | ||
Comment 3•10 years ago
|
||
Added patch to remove all disabled 10.6 machines from buildbot-configs.
Attachment #8762509 -
Flags: review?(kmoir)
Attachment #8762509 -
Flags: review?(coop)
| Assignee | ||
Comment 4•10 years ago
|
||
Only 30 slave machines for 10.6 are still enabled and seems to keep up with the load:
mysql> select count(*) from slaves where enabled=1 and name like 't-snow-r4%';
+----------+
| count(*) |
+----------+
| 30 |
+----------+
1 row in set (0.00 sec)
Comment 5•10 years ago
|
||
Keep in mind that your testing so far has been over the weekend before a workweek, so all you've kept up with were blocklist updates that only landed on two of the three branches running 10.6, and then a single push.
More reasonable load to plan for is a couple of pushes to each of the three branches, with relman sitting watching and waiting for results. And that you now absolutely unquestionably can't keep up with, because 30 slaves is exactly how many you need to run a set of tests on an opt build, and one short of how many you need to run a set of tests on a debug build. With 31 slaves, you could keep up with a single push to only one of the three branches at a time, only if the debug build finishes far enough ahead of the opt build to be done with tests before the opt build schedules its tests (and by looking at a couple of pushes, it looks like typically it doesn't, and there's a small bit of overlap), and only if nobody ever gets a 10.6 loaner and none of them ever break. But in your worst case, when someone pushes to try with 10.6 tests at just the wrong time right before a sheriff pushes a chemspill patch simultaneously to beta and release and esr45, you're looking at something close to 7 hours.
Comment 6•10 years ago
|
||
And a third of the still-enabled ones were already broken, or at least bent, having rotted away during the period of idleness since the load dropped. I think that's atypical and you don't need to plan on a third over the desired capacity, but I could well be wrong, not having seen how they behave around idleness other than over weekends.
| Assignee | ||
Comment 7•10 years ago
|
||
Enabled back 21 slaves,now there are 51 enabled slaves:
mysql> select count(*) from slaves where name like 't-snow-r4%' and enabled=1;
+----------+
| count(*) |
+----------+
| 51 |
+----------+
1 row in set (0.00 sec)
Thank you Phil for the warning,I will keep this number of slaves until some test will be performed and then we will have more informations about how many slaves can keep up with the load.
| Reporter | ||
Comment 8•10 years ago
|
||
Comment on attachment 8762509 [details] [diff] [review]
bug1279394_bb_1.patch
Review of attachment 8762509 [details] [diff] [review]:
-----------------------------------------------------------------
::: mozilla-tests/production_config.py
@@ +37,4 @@
> for i in range(102, 103): # Use win8's 102 for win10 // Bug 1191481
> SLAVES['win10']['t-w864-ix-%03i' % i] = {}
>
> +for i range(99, 115) + range(116, 129): # slaves 0034&0093&0094&0153 have been decommed ; bug 1279394 - 89 slaves have been decomed
The new range is fine, but you're missing the "in" in "for i in range()".
r+ with that fixed.
Attachment #8762509 -
Flags: review?(kmoir)
Attachment #8762509 -
Flags: review?(coop)
Attachment #8762509 -
Flags: review+
| Assignee | ||
Comment 9•10 years ago
|
||
A new patch was added,with a new range as more tests need to be run to see if the enabled slaves can keep up with the load therefore some of the slaves were reenabled.
Thank you Coop for review.
Attachment #8762509 -
Attachment is obsolete: true
Attachment #8762663 -
Flags: review?(coop)
| Reporter | ||
Updated•10 years ago
|
Attachment #8762663 -
Flags: review?(coop) → review+
Comment 10•10 years ago
|
||
Comment on attachment 8762663 [details] [diff] [review]
bug1279394_bb_1.patch
and merged to production
Attachment #8762663 -
Flags: checked-in+
| Assignee | ||
Updated•10 years ago
|
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Updated•8 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•6 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•