decommission (releng side) aws buildbot masters migrated to scl3

RESOLVED FIXED

Status

defect
RESOLVED FIXED
5 years ago
Last year

People

(Reporter: Callek, Assigned: Callek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

Assignee

Description

5 years ago
So, after moving mobile test masters to scl3 (rather than aws) and moving the once-aws test masters to scl3 [for in house machines] we certainly have a whole bunch of things we can remove.

:catlee suggests these IP's/hostnames are useful for mgerva who is needing to recreate a bunch of aws instances for masters. This will assist in needing less back<->forth with IT on flows/etc. So I'm explicitly not filing IT bugs for that stuff, but expect :mgerva/:catlee to wrangle those appropriate revokes after their work is all done.

Linux Test Masters:
* Can likely be reduced, but won't be in this bug.

Windows Test masters:
* Was aws masters: bm{69..74}
** Need to keep 1 in AWS for win64 tests on ec2
* TODO terminate 70..74
** Shut Down in Bug 971780 c#16

OSX Test masters:
* Was aws masters: bm{75..80}
* TODO terminate 75..80
** Shut Down in Bug 971780 c#16

Panda Test Masters:
* Was [never used] AWS masters: bm{90..94}
* TODO Terminate 90..94

Tegra Test masters:
* Was [never used] AWS masters: bm{95..98}
* TODO Terminate 95..98
Assignee

Updated

5 years ago
Summary: decomission (releng side) buildbot masters migrated to scl3 → decommission (releng side) aws buildbot masters migrated to scl3
Assignee

Comment 1

5 years ago
Disabled Termination protection and then Terminated in us-west2:

Instances: i-6c4b595e (buildbot-master72), i-7e4b594c (buildbot-master73), i-664b5954 (buildbot-master74), i-624b5950 (buildbot-master78), i-644b5956 (buildbot-master79), i-7a4b5948 (buildbot-master80), i-28bd981d (buildbot-master91), i-c86743fd (buildbot-master93), i-c6587cf3 (buildbot-master95), i-a6381c93 (buildbot-master97)

And the same for instances in us-east:

Instances: i-ec33a58f (buildbot-master98), i-fd254b9f (buildbot-master71), i-e01a2c89 (buildbot-master75), i-cb2c37ab (buildbot-master76), i-44bfe42a (buildbot-master77), i-af2d8ccd (buildbot-master90), i-00c29c6c (buildbot-master92), i-ce92c8a6 (buildbot-master94), i-a9fadfc6 (buildbot-master96)


Buildbot-master70 seems to have already been terminated, likely for mgerva's needs.
Assignee

Comment 2

5 years ago
This is likely a good thing anyway since mgerva will want different master config choices anyway if he reuses host names/etc
Attachment #8380930 - Flags: review?(catlee)
Assignee

Comment 3

5 years ago
bm90..98 had nagios notifications enabled, I just disabled them. Will need to make sure to re-enable in nagios-web UI if they come back as valid hosts. Or remove nagios-configs for them if they end up permanently dead.
Attachment #8380930 - Flags: review?(catlee) → review+
Assignee

Comment 4

5 years ago
Posted patch [puppet] revise nodes (obsolete) — Splinter Review
As long as we're leaving DNS and flows in place, I think we want to keep these available for puppetizing, but should certainly be removed when no longer existing.

That said, we certainly don't want to point at the master names we're removing in the tools step, which won't be wanted going forward anyway, so this patch reverts to a base node def
Attachment #8380939 - Flags: review?(catlee)
Assignee

Comment 5

5 years ago
this time with qref'ed
Attachment #8380940 - Flags: review?(catlee)
Assignee

Updated

5 years ago
Attachment #8380939 - Attachment is obsolete: true
Attachment #8380939 - Flags: review?(catlee)

Updated

5 years ago
Attachment #8380940 - Flags: review?(catlee) → review+
Assignee

Comment 6

5 years ago
https://hg.mozilla.org/build/puppet/rev/5aa2bc0d01d7
https://hg.mozilla.org/build/puppet/rev/bef8f4da9c7c

and

https://hg.mozilla.org/build/tools/rev/f6c525826ba0

All hosts are also annotated in slavealloc, so I think I'm done here.

Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)
Flags: needinfo?(mgervasini)
I have re-enabled buildbot-master70 (bug 966070), we need to understand if the new instance type is able to handle a build master workload. 

catlee, when the test is done, do we need more m3.medium build masters?
Flags: needinfo?(mgervasini) → needinfo?(catlee)
yes, we want to re-create all existing master instances as m3.medium
Flags: needinfo?(catlee)
Assignee

Comment 9

5 years ago
:mgerva, :catlee did we remove flows/etc as appropriate already?

"Leaving open and needinfo'ing to massimo to ensure the additional work identified in c#0 I'm purposely omitting is added to his plate. (before I close this bug)"
Flags: needinfo?(mgervasini)
Flags: needinfo?(catlee)
Hey Callek,

For bug 966070, we have used 12 hosts: bm{70..79}, bm91 and 94. Nothing else to needs to be done for them in this bug.

According to comment 0, the list of machines that needs to be decommissioned is now:
* buildbot-master69
* buildbot-master80
* buildbot-master90
* buildbot-master92
* buildbot-master93
* buildbot-master95
* buildbot-master96
* buildbot-master97
* buildbot-master98
Flags: needinfo?(mgervasini)
Assignee

Updated

5 years ago
Blocks: 990775
Assignee

Comment 11

5 years ago
Filed Bug 990775 and assigned to :mgerva
Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(catlee)
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.