976281 - decommission (releng side) aws buildbot masters migrated to scl3

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

So, after moving mobile test masters to scl3 (rather than aws) and moving the once-aws test masters to scl3 [for in house machines] we certainly have a whole bunch of things we can remove.

:catlee suggests these IP's/hostnames are useful for mgerva who is needing to recreate a bunch of aws instances for masters. This will assist in needing less back<->forth with IT on flows/etc. So I'm explicitly not filing IT bugs for that stuff, but expect :mgerva/:catlee to wrangle those appropriate revokes after their work is all done.

Linux Test Masters:
* Can likely be reduced, but won't be in this bug.

Windows Test masters:
* Was aws masters: bm{69..74}
** Need to keep 1 in AWS for win64 tests on ec2
* TODO terminate 70..74
** Shut Down in Bug 971780 c#16

OSX Test masters:
* Was aws masters: bm{75..80}
* TODO terminate 75..80
** Shut Down in Bug 971780 c#16

Panda Test Masters:
* Was [never used] AWS masters: bm{90..94}
* TODO Terminate 90..94

Tegra Test masters:
* Was [never used] AWS masters: bm{95..98}
* TODO Terminate 95..98

Comment 1

[Justin Wood (:Callek)]

Disabled Termination protection and then Terminated in us-west2:

Instances: i-6c4b595e (buildbot-master72), i-7e4b594c (buildbot-master73), i-664b5954 (buildbot-master74), i-624b5950 (buildbot-master78), i-644b5956 (buildbot-master79), i-7a4b5948 (buildbot-master80), i-28bd981d (buildbot-master91), i-c86743fd (buildbot-master93), i-c6587cf3 (buildbot-master95), i-a6381c93 (buildbot-master97)

And the same for instances in us-east:

Instances: i-ec33a58f (buildbot-master98), i-fd254b9f (buildbot-master71), i-e01a2c89 (buildbot-master75), i-cb2c37ab (buildbot-master76), i-44bfe42a (buildbot-master77), i-af2d8ccd (buildbot-master90), i-00c29c6c (buildbot-master92), i-ce92c8a6 (buildbot-master94), i-a9fadfc6 (buildbot-master96)


Buildbot-master70 seems to have already been terminated, likely for mgerva's needs.

Comment 2

[Justin Wood (:Callek)]

Attachment: [tools] remove terminated instances from production-masters

This is likely a good thing anyway since mgerva will want different master config choices anyway if he reuses host names/etc

Comment 3

[Justin Wood (:Callek)]

bm90..98 had nagios notifications enabled, I just disabled them. Will need to make sure to re-enable in nagios-web UI if they come back as valid hosts. Or remove nagios-configs for them if they end up permanently dead.

Comment 4

[Justin Wood (:Callek)]

Attachment: [puppet] revise nodes

As long as we're leaving DNS and flows in place, I think we want to keep these available for puppetizing, but should certainly be removed when no longer existing.

That said, we certainly don't want to point at the master names we're removing in the tools step, which won't be wanted going forward anyway, so this patch reverts to a base node def

Comment 5

[Justin Wood (:Callek)]

Attachment: [puppet] revise nodes

this time with qref'ed

Comment 6

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/puppet/rev/5aa2bc0d01d7
https://hg.mozilla.org/build/puppet/rev/bef8f4da9c7c

and

https://hg.mozilla.org/build/tools/rev/f6c525826ba0

All hosts are also annotated in slavealloc, so I think I'm done here.

Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)

Comment 7

[Massimo Gervasini [:massimo]]

I have re-enabled buildbot-master70 (bug 966070), we need to understand if the new instance type is able to handle a build master workload. 

catlee, when the test is done, do we need more m3.medium build masters?

Comment 8

[Chris AtLee [:catlee]]

yes, we want to re-create all existing master instances as m3.medium

Comment 9

[Justin Wood (:Callek)]

:mgerva, :catlee did we remove flows/etc as appropriate already?

"Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)"

Comment 10

[Massimo Gervasini [:massimo]]

Hey Callek,

For bug 966070, we have used 12 hosts: bm{70..79}, bm91 and 94. Nothing else to needs to be done for them in this bug.

According to comment 0, the list of machines that needs to be decommissioned is now:
* buildbot-master69
* buildbot-master80
* buildbot-master90
* buildbot-master92
* buildbot-master93
* buildbot-master95
* buildbot-master96
* buildbot-master97
* buildbot-master98

Comment 11

[Justin Wood (:Callek)]

Filed Bug 990775 and assigned to :mgerva