894398 - Re-purpose 24 out of 40 bld-linux64-ix machines as win64 builders

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Chris AtLee [:catlee]]

We're going to need more windows build capacity very soon. We currently have 24 ix machines that could be re-imaged into win64 build machines.

To preserve some minimal in-house capacity, we should keep at least 5 of these in-house. That means we could re-image 19.

I'll wait a few days for some bikeshedding, and then ask IT to reimage the boxes if there are no objections.

Comment 1

[Armen [:armenzg]]

+1

Comment 2

[Chris AtLee [:catlee]]

Ok, let's go ahead and re-image these guys.

Comment 3

[Armen [:armenzg]]

(In reply to Chris AtLee [:catlee] from comment #2)
> Ok, let's go ahead and re-image these guys.

Did you mean to move it to Relops?

Comment 4

[Armen [:armenzg]]

We have done this in the past in bug 860766.

We'll re-purpose 24 of them.

From try 11 of them (5 are left):
bld-linux64-ix-038
bld-linux64-ix-039
bld-linux64-ix-040
bld-linux64-ix-041
bld-linux64-ix-042
bld-linux64-ix-043
bld-linux64-ix-044
bld-linux64-ix-045
bld-linux64-ix-046
bld-linux64-ix-047
bld-linux64-ix-048
bld-linux64-ix-049
bld-linux64-ix-050

From build 13 of them (11 are left):
bld-linux64-ix-004
bld-linux64-ix-015
bld-linux64-ix-016
bld-linux64-ix-017
bld-linux64-ix-018
bld-linux64-ix-019
bld-linux64-ix-020
bld-linux64-ix-021
bld-linux64-ix-022
bld-linux64-ix-023
bld-linux64-ix-024
bld-linux64-ix-025
bld-linux64-ix-026


The new names should be:
w64-ix-slave134
w64-ix-slave135
w64-ix-slave136
w64-ix-slave137
w64-ix-slave138
w64-ix-slave139
w64-ix-slave140
w64-ix-slave141
w64-ix-slave142
w64-ix-slave143
w64-ix-slave144
w64-ix-slave145
w64-ix-slave146
w64-ix-slave147
w64-ix-slave148
w64-ix-slave149
w64-ix-slave150
w64-ix-slave151
w64-ix-slave152
w64-ix-slave153
w64-ix-slave154
w64-ix-slave155
w64-ix-slave156
w64-ix-slave157

Comment 5

[Armen [:armenzg]]

Attachment: win64.config.diff

Comment 6

[Armen [:armenzg]]

Attachment: win64.graphs.txt

Comment 7

[Armen [:armenzg]]

Attachment: win64.slavealloc.txt

Comment 8

[Armen [:armenzg]]

I'm purposely not putting any win64 machines on the try pool as we pretty much have all jobs finishing withing 15 minutes.

After the machines are imaged I will review these steps and see which ones still apply:
https://wiki.mozilla.org/ReferencePlatforms/Win64#Post-reimaging_steps

Comment 9

[Chris Cooper [:coop] (he/him)]

As mentioned in IRC today, I found these machines by accident when deploying the shutdown event tracker fix:

https://wiki.mozilla.org/ReferencePlatforms/Win64#Disable_shutdown_event_tracker

All these machines have that step performed on them (since it's brand new). I didn't check any of the other post-imaging steps.

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 785041 [details] [diff] [review]
win64.slavealloc.txt

I deployed this on slavealloc. The slaves are still disabled.

Comment 11

[Hal Wine [:hwine] use NI!]

Comment on attachment 785040 [details] [diff] [review]
win64.graphs.txt

lgtm

inserted into both stage & production databases

Comment 12

[Hal Wine [:hwine] use NI!]

Attachment: config changes for repurposed hosts

fix for off-by-one on start of ranges

Comment 13

[Hal Wine [:hwine] use NI!]

Comment on attachment 792521 [details] [diff] [review]
config changes for repurposed hosts

https://hg.mozilla.org/build/buildbot-configs/rev/75d2be427f83

All configuration work done (finally)

Comment 14

[Armen [:armenzg]]

Live in production.

Comment 15

[Armen [:armenzg]]

Attachment: fix.diff

This will make it 24 machines instead of 26.

Exception in /builds/buildbot/try1/master/twistd.log:
2013-08-22 09:00:03-0700 [Broker,28910,10.12.49.77] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.cred.error.UnauthorizedLogin:

Comment 16

[Armen [:armenzg]]

I disabled them in slavealloc with "bug 894398 - reboot once fixed buildbot-configs"

We also have to mark the hosts removed as "decomm" on slavealloc.

Comment 17

[Armen [:armenzg]]

Attachment: fix.diff

This is the right one.

Comment 18

[Hal Wine [:hwine] use NI!]

Comment on attachment 794120 [details] [diff] [review]
fix.diff

Review of attachment 794120 [details] [diff] [review]:
-----------------------------------------------------------------

Got it -- comment 4 said 11 from try, but listed 13. Clarification occurred in bug 900988 comment 3.

Comment 19

[Armen [:armenzg]]

(In reply to Armen Zambrano [:armenzg] (Use needinfo flag) (Release Enginerring) (EDT/UTC-4) from comment #17)
> Created attachment 794120 [details] [diff] [review]
> fix.diff

Live in production.

Comment 20

[Hal Wine [:hwine] use NI!]

moved existing hosts into aws regions in slavealloc.
added missing w64-ix-slave157
rebooted all
connecting to masters okay
done and done