744553 - camino-xserve4.scl3.mozilla.com has gone missing

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I noticed late last night that camino-xserve4 had 1) switched hostnames when reporting to tinderbox and 2) then gone yellow, indicating either the machine or mail had gone down.

Today when I had a chance to try and ssh in, I'm getting "ssh_exchange_identification: Connection closed by remote host", and if I try to ping it (from the jumphost), I get "Destination Host Unreachable".

Can someone please take a look and see what's wrong and, if necessary, reboot?  Thanks!  (Also, I'm not sure what component this belongs in these days with all the re-org, as it's been so long since we've had a malfunctioning tinderbox; someone please move it to the right component if necessary.)

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

FWIW, I'd forgotten all about bug 557145 and didn't check to see which NIC the box was using once it moved into the new colo; I suppose there's a chance that is the problem again.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, it's not even ARP-ing, so there's very little I can do about it from here in Chicago.  Thanks for the tip, though.  I didn't look at which NIC it was using before the move, either, but in the absence of indications otherwise, DC ops would use NIC 1.

Can you good folks have a look?

Comment 3

[Derek Moore [:dmoore]]

NIC1 on camino-xserve4 has seen better days. It's mechanically damaged, so it's easy for the network cable to become slightly dislodged and lose the electrical connection.

The cable is re-inserted and the server is back online.

We can move the uplink to NIC2, which is in better shape. Feel free to open another bug to schedule this.

Comment 4

[Derek Moore [:dmoore]]

Also, inventory has been updated with proper rack and switch port information.

Comment 5

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to Derek Moore from comment #3)
> NIC1 on camino-xserve4 has seen better days. It's mechanically damaged, so
> it's easy for the network cable to become slightly dislodged and lose the
> electrical connection.
> 
> The cable is re-inserted and the server is back online.

Thanks--especially on the quick turn-around!  I'm seeing builds again, as expected.

> We can move the uplink to NIC2, which is in better shape. Feel free to open
> another bug to schedule this.

Filed bug 744677 on that (hopefully that's the correct component).