Closed Bug 976281 Opened 8 years ago Closed 8 years ago

decommission (releng side) aws buildbot masters migrated to scl3

Categories

(Release Engineering :: General, defect)

x86_64
Windows 7
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Callek, Assigned: Callek)

References

Details

Attachments

(2 files, 1 obsolete file)

So, after moving mobile test masters to scl3 (rather than aws) and moving the once-aws test masters to scl3 [for in house machines] we certainly have a whole bunch of things we can remove.

:catlee suggests these IP's/hostnames are useful for mgerva who is needing to recreate a bunch of aws instances for masters. This will assist in needing less back<->forth with IT on flows/etc. So I'm explicitly not filing IT bugs for that stuff, but expect :mgerva/:catlee to wrangle those appropriate revokes after their work is all done.

Linux Test Masters:
* Can likely be reduced, but won't be in this bug.

Windows Test masters:
* Was aws masters: bm{69..74}
** Need to keep 1 in AWS for win64 tests on ec2
* TODO terminate 70..74
** Shut Down in Bug 971780 c#16

OSX Test masters:
* Was aws masters: bm{75..80}
* TODO terminate 75..80
** Shut Down in Bug 971780 c#16

Panda Test Masters:
* Was [never used] AWS masters: bm{90..94}
* TODO Terminate 90..94

Tegra Test masters:
* Was [never used] AWS masters: bm{95..98}
* TODO Terminate 95..98
Summary: decomission (releng side) buildbot masters migrated to scl3 → decommission (releng side) aws buildbot masters migrated to scl3
Disabled Termination protection and then Terminated in us-west2:

Instances: i-6c4b595e (buildbot-master72), i-7e4b594c (buildbot-master73), i-664b5954 (buildbot-master74), i-624b5950 (buildbot-master78), i-644b5956 (buildbot-master79), i-7a4b5948 (buildbot-master80), i-28bd981d (buildbot-master91), i-c86743fd (buildbot-master93), i-c6587cf3 (buildbot-master95), i-a6381c93 (buildbot-master97)

And the same for instances in us-east:

Instances: i-ec33a58f (buildbot-master98), i-fd254b9f (buildbot-master71), i-e01a2c89 (buildbot-master75), i-cb2c37ab (buildbot-master76), i-44bfe42a (buildbot-master77), i-af2d8ccd (buildbot-master90), i-00c29c6c (buildbot-master92), i-ce92c8a6 (buildbot-master94), i-a9fadfc6 (buildbot-master96)


Buildbot-master70 seems to have already been terminated, likely for mgerva's needs.
This is likely a good thing anyway since mgerva will want different master config choices anyway if he reuses host names/etc
Attachment #8380930 - Flags: review?(catlee)
bm90..98 had nagios notifications enabled, I just disabled them. Will need to make sure to re-enable in nagios-web UI if they come back as valid hosts. Or remove nagios-configs for them if they end up permanently dead.
Attachment #8380930 - Flags: review?(catlee) → review+
Attached patch [puppet] revise nodes (obsolete) — Splinter Review
As long as we're leaving DNS and flows in place, I think we want to keep these available for puppetizing, but should certainly be removed when no longer existing.

That said, we certainly don't want to point at the master names we're removing in the tools step, which won't be wanted going forward anyway, so this patch reverts to a base node def
Attachment #8380939 - Flags: review?(catlee)
this time with qref'ed
Attachment #8380940 - Flags: review?(catlee)
Attachment #8380939 - Attachment is obsolete: true
Attachment #8380939 - Flags: review?(catlee)
Attachment #8380940 - Flags: review?(catlee) → review+
https://hg.mozilla.org/build/puppet/rev/5aa2bc0d01d7
https://hg.mozilla.org/build/puppet/rev/bef8f4da9c7c

and

https://hg.mozilla.org/build/tools/rev/f6c525826ba0

All hosts are also annotated in slavealloc, so I think I'm done here.

Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)
Flags: needinfo?(mgervasini)
I have re-enabled buildbot-master70 (bug 966070), we need to understand if the new instance type is able to handle a build master workload. 

catlee, when the test is done, do we need more m3.medium build masters?
Flags: needinfo?(mgervasini) → needinfo?(catlee)
yes, we want to re-create all existing master instances as m3.medium
Flags: needinfo?(catlee)
:mgerva, :catlee did we remove flows/etc as appropriate already?

"Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)"
Flags: needinfo?(mgervasini)
Flags: needinfo?(catlee)
Hey Callek,

For bug 966070, we have used 12 hosts: bm{70..79}, bm91 and 94. Nothing else to needs to be done for them in this bug.

According to comment 0, the list of machines that needs to be decommissioned is now:
* buildbot-master69
* buildbot-master80
* buildbot-master90
* buildbot-master92
* buildbot-master93
* buildbot-master95
* buildbot-master96
* buildbot-master97
* buildbot-master98
Flags: needinfo?(mgervasini)
Blocks: 990775
Filed Bug 990775 and assigned to :mgerva
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(catlee)
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.