1141890 - decom socorro2.dev.dmz.phx1

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[:Atoll]

We removed socorro1.dev.dmz.phx1 last year in bug 968591, which caused bug 1042269. I'd like to fix the latter bug by decom's socorro1.

:rhelmer approves, setting needinfo? :lonnen and :phrawzty to signoff so we can proceed.

Comment 1

[Daniel Maher [:phrawzty]]

This machine has been invalid since socorro1.dev.dmz.phx1 was decom'd.

+1

Comment 2

[Lonnen :lonnen]

r+ decomm away

Comment 3

[Vinh Hua [:vinh]]

1) socorro2.dev.dmz.phx1.mozilla.com has address 10.8.74.71

2) removed from nagios
vhua$ svn diff
Index: phx1.pp
===================================================================
--- phx1.pp	(revision 101974)
+++ phx1.pp	(working copy)
@@ -1839,14 +1839,6 @@
                 'socorro-admin',
             ]
         },
-        'socorro2.dev.dmz.phx1.mozilla.com' => {
-            parents => 'esx-cluster1.ops.phx1.mozilla.com',
-            contact_groups => 'socorroalerts',
-            hostgroups => [
-                'generic-preprod',
-                'virtual'
-            ]
-        },
         'sp-processor01.phx1.mozilla.com' => {
             parents => 'boa-c08-a1.console.phx1.mozilla.com',
             hostgroups => [

Committed revision 102023.

3) no external monitoring

4) no netvault, no nfs
[[e-mail]1 ~]$ df
Filesystem            1K-blocks       Used  Available Use% Mounted on
/dev/sda3             306436632    7273128  283599884   3% /
tmpfs                   4030780          0    4030780   0% /dev/shm
/dev/sda1                991512      54836     885476   6% /boot
10.8.74.13:/symbols 12884901888 9293367424 3591534464  73% /mnt/socorro/symbols

5) puppet disabled
[[e-mail]1 ~]$ sudo puppetctl disable -d +365d
Puppet has been disabled by vhua at 2015-03-12 13:51 until 2016-03-11 13:51

Comment 4

[Vinh Hua [:vinh]]

6) host powered off, waiting 1 week for any complaints

[[e-mail]1 ~]$ sudo shutdown -P now

Broadcast message from [e-mail]
	(/dev/pts/0) at 13:54 ...

The system is going down for power off NOW!
[[e-mail]1 ~]$ Connection to socorro2.dev.dmz.phx1.mozilla.com closed by remote host.
Connection to socorro2.dev.dmz.phx1.mozilla.com closed.

Comment 5

[Greg Cox [:gcox]]

> 4) no netvault, no nfs
> 10.8.74.13:/symbols 12884901888 9293367424 3591534464  73%
> /mnt/socorro/symbols

That's NFS.  Noted for next week.

Comment 6

[Vinh Hua [:vinh]]

:gcox - good catch, thanks!

Comment 7

[Vinh Hua [:vinh]]

7) host has remained off
vhua$ ping 10.8.74.71
PING 10.8.74.71 (10.8.74.71): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

8) DNS records deleted

9) unable to delete inventory record (separate issue)

10) don't think it's part of a cluster or behind load balancer

11) deleted in RHN 

12) deleted from puppet dashboard

Comment 8

[Vinh Hua [:vinh]]

13) no node definition to remove from puppet

14) open bug 1147114  for ACL removal


:gcox - Once item #14 clears, delete away please.

Comment 9

[Greg Cox [:gcox]]

NFS acl pulled.  (I don't need to be blocked by the netops bug, we're already in destruct mode).

Comment 10

[Greg Cox [:gcox]]

VM deleted from disk after PM's with :vinh.

Comment 11

[Vinh Hua [:vinh]]

ATM inventory records could not be purged. I've verified with :sal and :van on another host.  I've updated the notes for decom socorro2.dev.dmz.phx1