773682 - kvms in scl3 and mtv1 went down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ashish Vijayaram [:ashish]]

09:16:42 < nagios-rele> | [73] kvm1.private.releng.scl3 is DOWN: PING 
CRITICAL - Packet loss = 100%

Aj and arr are working on moving VMs to a different node

Comment 1

[Adrian J Fernandez [:Aj]]

General update:

The servers are up.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Amy is working on repairing problems on the hosts themselves.  The interface issue itself is resolved.

Comment 4

[Amy Rich [:arr] [:arich]]

This issue impacted any kvm nodes that were controlled by sysadmins puppet.  Errors in the bond0 (and in at least one case br0) config files took down the bond network interface for machines as they ran puppet.

A fix was checked in to puppet, and hosts that had dropped off the network were fixed manually and brought back online.

Once the machines were reachable again, I did the following for the releng kvm nodes:

if the master node went down, logged into a different node and ran:

gnt-cluster master-failover

reactivated the sets of degraded disks:

for i in `gnt-instance list| awk '{print $1}'`|grep -v Instance; do echo $i; done

rebuilt the secondary disks for each node in case they became out of sync while the two kvm nodes were disconnected:

for i in `gnt-instance list| awk '{print $1}'|grep -v Instance`; do gnt-instance replace-disks -s $i; done

The latter is still running and will take some time (probably 5+ hours) as it cycles through all of them vms.  I have downtimed the drbd checks on the releng kvm servers in scl3 and mtv1 for 5 hours for now.

Comment 5

[Aki Sasaki (not active)]

How do we avoid this in the future?

Comment 6

[Mike Taylor [:bear]]

notes:
 - trees closed at 0938
 - trees opened at 1051

Comment 7

[Amy Rich [:arr] [:arich]]

ALso, this impacted more than just releng.  Metrics also lost machines, and I'm not sure if anyone in metrics/IT has reconstructed the disks on those vms yet.

Comment 8

[Adrian J Fernandez [:Aj]]

As far as I know, it only affected kvm1.generic.metrics.scl3 and no other alerts came in.

Perhaps :cyliang could verify if everything looks ok on their end?

Comment 9

[Amy Rich [:arr] [:arich]]

The secondary rebuilds of all releng kvm vms in mtv1 and scl3 have completed successfully.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Aki Sasaki [:aki] from comment #5)
> How do we avoid this in the future?

Who can best answer this?

Comment 11

[Corey Shields [:cshields]]

(In reply to John O'Duinn [:joduinn] from comment #10)
> (In reply to Aki Sasaki [:aki] from comment #5)
> > How do we avoid this in the future?
> 
> Who can best answer this?

I can, but I don't have the answer yet.

Comment 12

[Corey Shields [:cshields]]

(In reply to Aki Sasaki [:aki] from comment #5)
> How do we avoid this in the future?

More puppet testing in a stage environment..  Something we can't easily resource right now.  This was caused by a one-off puppet problem - not one we can easily build to prevent specifically in the future.

KVM has been stabilized, closing this out.