1244790 - decomission vcssync1 & vcssync2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Hal Wine [:hwine] use NI!]

Services are now running from developer services AWS account - these are no longer needed.

Steps needed:
 - remove from Nagios monitoring
 - terminate instances
 - delete storage

Comment 1

[Hal Wine [:hwine] use NI!]

Attachment: decom-vcssync.patch

Pull monitoring from vcssync{1,2}

Some checks appear to have _only_ been for these boxes, so want verification that pulling the whole check was correct. (disk-/ & disk-/opt)

Comment 2

[Amy Rich [:arr] [:arich]]

Comment on attachment 8714391 [details] [diff] [review]
decom-vcssync.patch

Review of attachment 8714391 [details] [diff] [review]:
-----------------------------------------------------------------

I think you've captured everything correctly. Let me know when you check stuff in, and I can force an svn up and puppet run on the nagios server.

Comment 3

[Hal Wine [:hwine] use NI!]

Comment on attachment 8714391 [details] [diff] [review]
decom-vcssync.patch

$ svn commit
Sending        modules/nagios/manifests/releng/hostgroups.pp
Sending        modules/nagios/manifests/releng/scl3.pp
Sending        modules/nagios/manifests/releng/servicegroups.pp
Sending        modules/nagios/manifests/releng/services.pp
Transmitting file data ....
Committed revision 114602.

Comment 4

[Hal Wine [:hwine] use NI!]

this appears to have deployed correctly, no longer see these hosts listed in nagios.

Comment 5

[Hal Wine [:hwine] use NI!]

Nagios alerts are gone, someone with AWS chops needs to do rest.

Comment 6

[Hal Wine [:hwine] use NI!]

(aws_manager)[[e-mail] cloud-tools]$ aws_manage_instances status vcssync{1..2}
2016-02-18 16:52:34,896 - INFO - Found vcssync1 (i-c38fc2f7)...
   Name: vcssync1
     ID: i-c38fc2f7
     IP: 10.132.48.15
Enabled: True
  State: stopped
   Tags: FQDN -> vcssync1.srv.releng.usw2.mozilla.com, moz-state -> ready, moz-type -> vcssync, Name -> vcssync1, created -> 2013-10-10 21:25:42 PST
========================================================================
2016-02-18 16:52:34,914 - INFO - Found vcssync2 (i-b0d76287)...
   Name: vcssync2
     ID: i-b0d76287
     IP: 10.132.49.15
Enabled: True
  State: stopped
   Tags: Name -> vcssync2, moz-type -> vcssync, created -> 2013-10-10 21:39:15 PST, FQDN -> vcssync2.srv.releng.usw2.mozilla.com, moz-state -> ready
========================================================================

Comment 7

[Justin Wood (:Callek)]

I just disabled termination protection and marked both to be terminated....

Comment 8

[Hal Wine [:hwine] use NI!]

Can you check if they are fully gone now, please?  If so, r/f the bug and gather endorphins!

Comment 9

[Justin Wood (:Callek)]

I have the gut feeling there may be some ebs lying dormant from these hosts, but I couldn't find a good way to correlate.

Anyway, the host instances themselves are gone.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Deleted A and PTR records from inventory.