1383266 - decom a couple scl3 windows tests masters

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P4)

Description

[Kim Moir [:kmoir] ET]

We should evaluate load, but may be able to decomm some scl3 windows test masters after we have migrated some win32/win64 tests to tc as tier1 on trunk next week.

Here is the current list
buildbot-master69.bb.releng.use1.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master109.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master110.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master111.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master112.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master119.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master126.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master127.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Will hold on to grab this bug until the EOW when we can evaluate how the TC migration went. That's due on Wednesdays.

Comment 2

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

So we have for buildbot windows:
-150 t-w732-ix
-345 t-w864-ix
-40 t-xp32-ix (and only first 10 are enabled)

From the list above mentiond by Kim we use buildbot-master69 for test purpose,anyway this master was not used in production.
We have for the moment 7 bm masters,we can reduce this number to 3 or less,maybe keeping only :

buildbot-master109.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master110.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master
buildbot-master111.bb.releng.scl3.mozilla.com:/builds/buildbot/tests1-windows/master

Comment 3

[Amy Rich [:arr] [:arich]]

We also have t-1064-ix in scl3, so we should make sure we have enough capacity for those.

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

yes sorry also 75  t-1064-ix machines,I forgot to mention them,thank you Amy

Comment 5

[Kim Moir [:kmoir] ET]

That would probably be fine, if you want to be super cautious you could remove two, re-evaluate load and then see if you can remove another.  As a side note, Joel wants to enable some more buildbot tests in bug 1393198

Comment 6

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Will keep for the moment bm109 ,bm110 and bm111.

mysql> select masters.nickname, count(*), masters.enabled from slaves, masters where masters.nickname like '%windows%' and slaves.current_masterid = masters.masterid group by masters.nickname;
+----------------------+----------+---------+
| nickname             | count(*) | enabled |
+----------------------+----------+---------+
| bm109-tests1-windows |       86 |       1 |
| bm110-tests1-windows |       84 |       1 |
| bm111-tests1-windows |       82 |       1 |
| bm112-tests1-windows |       81 |       1 |
| bm119-tests1-windows |       81 |       1 |
| bm126-tests1-windows |       81 |       1 |
| bm127-tests1-windows |       80 |       1 |

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1383266_puppet.patch

Puppet patch to remove bm112,bm119,bm126,bm127

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1383266_tools.patch

Remove from tools repository

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1383266_int.patch

Disable nagios alerts for BB masters

Comment 10

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Disabled yesterday these 4 bm masters and the masters left seems to handle the machines left on window:

+----------------------+----------+---------+
| nickname             | count(*) | enabled |
+----------------------+----------+---------+
| bm109-tests1-windows |      191 |       1 |
| bm110-tests1-windows |      189 |       1 |
| bm111-tests1-windows |      187 |       1 |
| bm128-tests1-windows |      134 |       1 |
| bm129-tests1-windows |      134 |       1 |
| bm137-tests1-windows |      133 |       1 |
| bm138-tests1-windows |      133 |       1 |
| bm139-tests1-windows |      133 |       1 |
| bm140-tests1-windows |      133 |       1 |
+----------------------+----------+---------+
 We had a spike of 2k pending jobs on w732 but only for 15 minutes so I will continue with the decom.

Comment 11

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Also deleted from slavealloc.

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8901153 [details] [diff] [review]
bug1383266_tools.patch

https://hg.mozilla.org/build/tools/rev/506b046da7bb

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Note explaining the priority level: P4 doesn't mean we've lowered the priority, but the contrary. However, we're aligning these levels to the buildduty quarterly deliverables, where P1-P3 are taken by our daily waterline KTLO operational tasks.

Comment 14

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

All done here.