1397225 - reimage 20 win8 machines as windows 10

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to move 20 machines from win8->win10 to accommodate load of moving mochitest-chrome/clipboard/gpu jobs to win10.

My thought is that moving 20 jobs is enough to not starve win8  (the backlog there is zero most of the time) and if we find after the move that the backlog remains at or near zero then we could move another 10 machines.

Comment 1

[Amy Rich [:arr] [:arich]]

jmaher: I intend to use 352 as one of the machines we convert. THat's listed as a loaner for you. Is it okay to reimage as w10 and put back into the pool?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

yes, let me update the bug on the loaners as well.

Comment 3

[Amy Rich [:arr] [:arich]]

I have disabled the following machines in preparation for them becoming the new w10 machines:

t-w864-ix-337.wintest.releng.scl3.mozilla.com
t-w864-ix-338.wintest.releng.scl3.mozilla.com
t-w864-ix-339.wintest.releng.scl3.mozilla.com
t-w864-ix-340.wintest.releng.scl3.mozilla.com
t-w864-ix-341.wintest.releng.scl3.mozilla.com
t-w864-ix-342.wintest.releng.scl3.mozilla.com
t-w864-ix-343.wintest.releng.scl3.mozilla.com
t-w864-ix-344.wintest.releng.scl3.mozilla.com
t-w864-ix-345.wintest.releng.scl3.mozilla.com
t-w864-ix-346.wintest.releng.scl3.mozilla.com
t-w864-ix-347.wintest.releng.scl3.mozilla.com
t-w864-ix-348.wintest.releng.scl3.mozilla.com
t-w864-ix-349.wintest.releng.scl3.mozilla.com
t-w864-ix-350.wintest.releng.scl3.mozilla.com
t-w864-ix-351.wintest.releng.scl3.mozilla.com
t-w864-ix-352.wintest.releng.scl3.mozilla.com
t-w864-ix-353.wintest.releng.scl3.mozilla.com
t-w864-ix-354.wintest.releng.scl3.mozilla.com
t-w864-ix-355.wintest.releng.scl3.mozilla.com
t-w864-ix-356.wintest.releng.scl3.mozilla.com

They will become:

t-w1064-ix-076.wintest.releng.scl3.mozilla.com
t-w1064-ix-077.wintest.releng.scl3.mozilla.com
t-w1064-ix-078.wintest.releng.scl3.mozilla.com
t-w1064-ix-079.wintest.releng.scl3.mozilla.com
t-w1064-ix-080.wintest.releng.scl3.mozilla.com
t-w1064-ix-081.wintest.releng.scl3.mozilla.com
t-w1064-ix-082.wintest.releng.scl3.mozilla.com
t-w1064-ix-083.wintest.releng.scl3.mozilla.com
t-w1064-ix-084.wintest.releng.scl3.mozilla.com
t-w1064-ix-085.wintest.releng.scl3.mozilla.com
t-w1064-ix-086.wintest.releng.scl3.mozilla.com
t-w1064-ix-087.wintest.releng.scl3.mozilla.com
t-w1064-ix-088.wintest.releng.scl3.mozilla.com
t-w1064-ix-089.wintest.releng.scl3.mozilla.com
t-w1064-ix-090.wintest.releng.scl3.mozilla.com
t-w1064-ix-091.wintest.releng.scl3.mozilla.com
t-w1064-ix-092.wintest.releng.scl3.mozilla.com
t-w1064-ix-093.wintest.releng.scl3.mozilla.com
t-w1064-ix-094.wintest.releng.scl3.mozilla.com
t-w1064-ix-095.wintest.releng.scl3.mozilla.com

aobreja: can you please add the necessary configs to support these in buildbot?

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

sure,I'm on it

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1397225_bb.patch

Patch for buildbot-config

Comment 6

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1397225_slavealloc.csv

Adding the slaves to slavealloc.

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Added the new slaves to slavealloc

mysql> select slaveid, name from slaves where name like 't-w1064-ix%' order by cast(substring(name, -3) as unsigned) desc limit 20;
+---------+----------------+
| slaveid | name           |
+---------+----------------+
|   34325 | t-w1064-ix-095 |
|   34323 | t-w1064-ix-094 |
|   34321 | t-w1064-ix-093 |
|   34319 | t-w1064-ix-092 |
|   34317 | t-w1064-ix-091 |
|   34315 | t-w1064-ix-090 |
|   34313 | t-w1064-ix-089 |
|   34311 | t-w1064-ix-088 |
|   34309 | t-w1064-ix-087 |
|   34307 | t-w1064-ix-086 |
|   34305 | t-w1064-ix-085 |
|   34303 | t-w1064-ix-084 |
|   34301 | t-w1064-ix-083 |
|   34299 | t-w1064-ix-082 |
|   34297 | t-w1064-ix-081 |
|   34295 | t-w1064-ix-080 |
|   34293 | t-w1064-ix-079 |
|   34291 | t-w1064-ix-078 |
|   34289 | t-w1064-ix-077 |
|   34287 | t-w1064-ix-076 |
+---------+----------------+
20 rows in set (0.00 sec)

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Also removed the missing entries for the win8 slaves that were migrated to other win10 pool.

mysql> delete from slaves where (name like 't-w864-ix%' and convert(substring(name,-3), unsigned integer) > 336 and convert(substring(name,-3), unsigned integer) < 357);
Query OK, 20 rows affected (0.01 sec)


mysql> select slaveid, name from slaves where name like 't-w864-ix%' order by cast(substring(name, -3) as unsigned) desc limit 5;
+---------+---------------+
| slaveid | name          |
+---------+---------------+
|   34012 | t-w864-ix-336 |
|   34010 | t-w864-ix-335 |
|   34008 | t-w864-ix-334 |
|   34006 | t-w864-ix-333 |
|   34004 | t-w864-ix-332 |
+---------+--------------

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8905015 [details] [diff] [review]
bug1397225_bb.patch

https://hg.mozilla.org/build/buildbot-configs/rev/9636990ec3e7a4c8addce1f35c20153360e45119

Comment 10

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

The only thing that remain is to rename and re-image the 20 machines that were moved from Windows 8 pool to Windows 10 pool,the change must be done in Nagios and Inventory,Amy can you do this change,on buildbot the configs are done?

Comment 11

[Amy Rich [:arr] [:arich]]

inventory changes made via invtool
nagios changes checked in with: commit 4420faa8b0fbf57b6274738e4c7b902ae776d36a
and I kicked off the reimages

aobreja: can you validate the installs in an hour/hour and a half or so and enable them if they're ready? I seem to recall that they might need an extra reboot after they're done installing and enabled.

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I will be around for another hour so if it's done until then I will check if not Sebastian will still be available and can check,thank you Amy.

Comment 13

[Dave Williams [:daveio]]

The nagios changes have broken the config - can they please be fixed. Let me know if you need any help.

Comment 14

[Dave Williams [:daveio]]

In the meantime I have rolled back the changes to get releng Nagios up and running again.

Comment 15

[Amy Rich [:arr] [:arich]]

There weren't any alerts in irc that I saw. What was the error? Rolling back is also going to cause errors since those hosts aren't in DNS anymore.

Comment 16

[Amy Rich [:arr] [:arich]]

The problem was a backtic in the config file. daveio said he checked in the corrected file.

Comment 17

[Sebastian Pacurar [:spacurar]]

Just checked if the machines have windows 10 as their os, and noticed that 87, 91, 92 ,93 and 94 still run under windows 8. Currently reimaging those ones.

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Checked all machines and all seems to be fine and are running under win10,I also enabled them.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

is this resolved so I could move test jobs over?

Comment 20

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I thing everything is done here,but better ask Amy to be on the safe side.