1249868 - crashplan2.private.tor1.mozilla.com doesn't come back up

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[Ludovic Hirlimann [:Usul]]

https://inventory.mozilla.org/en-US/systems/show/4872/

I've patched the machine but it doesn't come back up on reboot.

Comment 1

[Ludovic Hirlimann [:Usul]]

Can't seeh to the oob nor ipmitool to it :

[[e-mail]1 ~]$  ipmitool -H 10.242.6.177 -U BBBBB -P CCCCC sensor
Error: Unable to establish LAN session

Comment 2

[Ludovic Hirlimann [:Usul]]

Sat 05:35:06 PST [5036] crashplan2.private.tor1.mozilla.com (10.242.75.101) is UP :PING OK - Packet loss = 0%, RTA = 90.13 ms

but still can't connect to it.

Comment 3

[Peter Radcliffe [:pir]]

jlin took a look and reported this on the console:
  e1000e 0000:03:00.0: eth0 reset adapter unexpectedly / Timesync Tx Control register not set as expected
and network was going down regularly even after reboots.

Googling for that led me to:
https://bugs.centos.org/view.php?id=6810

I tried this, quickly after a reboot:
  sudo sed -i.pir 's]^\(.*kernel /vml.*\)$]\1 pcie_aspm=off]' /boot/grub/grub.conf 

and rebooted again. It seems stable for now, jlin is monitoring.

Comment 4

[Peter Radcliffe [:pir]]

:digi have you seen this before?

Comment 5

[Jonathan Lin - Not watching NI/offline [:jlin] [[e-mail]]]

it's been responding to pings and I've been logged in via ssh for about 20 min - should be fine now. Thanks Pir and Usul!

Comment 6

[Brian Hourigan [:digi]]

(In reply to Peter Radcliffe [:pir] from comment #4)
> :digi have you seen this before?

I have never seen this before.

Comment 7

[Jonathan Lin - Not watching NI/offline [:jlin] [[e-mail]]]

https://bugs.centos.org/view.php?id=6810#c19026 - also suggests turning the "Active State Power Management" bios option to "enabled".

if the network on this server drops off again, I could investigate that on the next reboot.

Comment 8

[Ryan C [:ryanc] (UTC-4)]

Still looks good. Reopen if this happens again.