757343 - cb-seamonkey-linux-01 has failed to recover from reboot

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Edmund Wong (:ewong)]

Once again, cb-seamonkey-linux-01 has taken leave without returning
to duty.  

It needs to be given another swift kick.

Comment 1

[Ashish Vijayaram [:ashish]]

Assigning to dustin, who'd kicked it a few days back.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Edmund, have you tried re-running mkfs on /builds?

Also, how are you rebooting this machine?

What kind of filesystem is /builds?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I ran mkfs.ext3 on /dev/hdc1, so with luck this should not continue to fail this way.

This is a journaled filesystem, but I suspect that parallels is doing some write re-ordering that's causing the journal to be inconsistent on reboot.  If you do reboot in the future, try to unmount /builds manually first -- kill everything using it first.

Comment 4

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> I ran mkfs.ext3 on /dev/hdc1, so with luck this should not continue to fail
> this way.
> 
> This is a journaled filesystem, but I suspect that parallels is doing some
> write re-ordering that's causing the journal to be inconsistent on reboot. 
> If you do reboot in the future, try to unmount /builds manually first --
> kill everything using it first.

Buildbot itself also reboots the system, with a simple |sudo reboot|. So manually unmounting/killing everything on it is infeasable :(