444700 - qm-pmac-trunk06 needs a reboot

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[alice nodelman [:alice] [:anode]]

Not reachable by ssh/vnc.

Comment 1

[Justin Fitzhugh]

can we troubleshoot why this keeps happening to help with the frequent reboot bugs?  why are these a) rebooting so often and b) not coming back?  think you have local mini's to replicate the issue...no?

Comment 2

[alice nodelman [:alice] [:anode]]

qm-pmac-trunk04/05/06 are known frequent offenders in terms of loss of access to vnc/ssh.  We've tried a couple of solutions so far:

- updating the version of leopard running on the machine
- ensuring that the machine doesn't go into an unwakeup-able suspended state.

Hard to believe, but that has decreased the amount of reboots required, but it is still unreasonably high for these machines.

Comment 3

[matthew zeier [:mrz]]

Any chance they just overheat and lock up?  Anything in the logs on that?

Comment 4

[alice nodelman [:alice] [:anode]]

Where would such a log be?

I won't be able to access the machine to check any logs till it gets a kick.

Comment 5

[matthew zeier [:mrz]]

On OSX, /var/log/system.log .  

Comment 6

[alice nodelman [:alice] [:anode]]

Is this scheduled for reboot?  Or fallen through the cracks...

Comment 7

[Phong Tran [:phong]]

I will head down after the meeting to reboot it.

Comment 8

[Phong Tran [:phong]]

I forgot to update this bug.  qm-pmac-trunk06 was rebooted yesterday.

Comment 9

[alice nodelman [:alice] [:anode]]

I don't quite know what to look for in the system.log - but I wasn't seeing anything in there connected to 'heat' or 'temp'.  It also doesn't have anything that looks like the obvious failure point to me.  

Also, the settings for the OS where correct (no sleeping, no hibernates, etc).