438811 - bm-xserve11 wont boot

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Nick Thomas [:nthomas] (UTC+12)]

bm-xserve11 does debug builds for Firefox 3.0 on mac, so it's on the Firefox tinderbox tree. It's normally solid as a rock but the build starting at 2008/06/11 23:25 PDT never completed (normally takes 10 minutes). There were no checkins so this is looking like a random glitch or hardware problem.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Can't get a login prompt with ssh despite the initial connection being made, and doesn't respond to VNC either.

Over to Server Ops for someone to look at the console. It's a Tier 1 box but the tree is closed at the moment, so no need to set blocker severity.

Comment 2

[matthew zeier [:mrz]]

Console was hung (all keyboard LEDs lit up but nothing on the monitor).  Power cycled.  Box is up at login prompt.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Looks like it was doing a checkout when it went boom. Saved at
 /builds/tinderbox/Fx-Trunk-test_mem/Darwin_8.8.4_Depend/Darwin_8.8.4_Depend.log.20080611-hang

Tinderbox restarted.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Gah, it did it again. Could you do a disk check please ?

Comment 5

[matthew zeier [:mrz]]

called for remote hands to power cycle.

Comment 6

[matthew zeier [:mrz]]

power cycled but it's down again? 

Comment 7

[matthew zeier [:mrz]]

I can keep power cycling but it keeps going dark, usually after you start doing something.  Something to fix on your end?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Maybe. Tinderbox says a build started at 2008/06/12 09:49, so that's comment #3 & 4. As far as I can tell no-one touched it after that, but CC'ing people that might've.

If it was touched then we should definitely clobber the build and have a general poke around. If not, then I think we're requesting some hardware diagnostics and a disk verify. 

Comment 9

[[e-mail] (:bhearsum)]

I haven't touched it

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Ok, I'm on the hook from the build side. Please reboot this again when you can get hold of me on IRC, then I can look at it immediately.

Comment 11

[matthew zeier [:mrz]]

remote hands called.

Comment 12

[matthew zeier [:mrz]]

Box rebooted.

Comment 13

[matthew zeier [:mrz]]

Host won't boot off local disks.

Comment 14

[matthew zeier [:mrz]]

Nick - OS reload?  

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

It's not too bad if we have to put the 10.4 Intel image on there and set up tinderbox again, but I'd really like to be confident in the hardware first.

Can we boot of a diagnostic CD and try to figure out if it's a problem with the RAID setup or disk failure or some other hardware fault? If one disk has died then we could replace that disk and rebuild the RAID. If the RAID is corrupted, could we look at the SMART status on the disks in case there's a failure right around the corner. 

Comment 16

[matthew zeier [:mrz]]

Over to Phong to check.

Comment 17

[Phong Tran [:phong]]

Verifying volume "untitled RAID Set 1"
Checking HFS Plus Volume
Checking Extents Overflows file.
Checking Catalog file.
Invalid extent entry
Incorrect block count for file Compose.strings
(It should be 0 instead of 49247)
Invalid extent entry
Checking multi-linked files.
Checking catalog hierarchy
Checking Extent Attributes file.
Checking volume bitmap.
Volume Bit Map needs minor repair.
Checking volume information.
Invalid volume free block count.
(It should be 13608976 instead of 13608963).
The volume Server RAID needs to be repaired.

Error: The underlying task reported failure on exit

1 HFS volume checked
	Volume needs repair

Comment 18

[Phong Tran [:phong]]

volume was successfully repaired, but it still won't boot to OS.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

What else can we try here, to get this machines back online?

Now that the disk has been repaired, maybe we could try imaging from another similar xserve, instead of imaging from a clean OSinstall?

Comment 20

[Phong Tran [:phong]]

Do you have a server I can take down to create an image from?

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Please use Build's "gold image" for Intel 10.4 rather than a running machine.

Comment 22

[Rob Campbell [:rc] (:robcee)]

if needed, we can return qm-xserve06 for re-imaging. It's redundant on the unittest farm.

Comment 23

[Phong Tran [:phong]]

I'll try the 10.4 image first.

Comment 24

[Phong Tran [:phong]]

bm-xserve11 has been imaged with gold 10.4 image from bm-xserve09.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Thanks Phong, looks good. Taking back to RelEng for tinderbox setup.

I've set the hostname to bm-xserve11.build.m.o. Coop, anything special for a debug tinderbox setup ? Do you have time to get this going again ?

Comment 26

[Reed Loden [:reed]]

This is a tier 1 machine on the 1.9.0 branch... why isn't the tree closed?

Comment 27

[Reed Loden [:reed]]

(In reply to comment #26)
> This is a tier 1 machine on the 1.9.0 branch... why isn't the tree closed?

Tree is now closed.

Comment 28

[Mike Beltzner [:beltzner, not reading bugmail]]

This is blocking our work on Gecko 1.9.0.1, and should be considered a P1 blocker. To whom should it be assigned?

Comment 29

[Chris Cooper [:coop] (he/him)]

A build is in progress now, but we're running in one-off mode until I'm sure the config is right. Don't let me forget to get a multi-config.pl setup!

Comment 30

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #29)
> Don't let me forget to get a multi-config.pl setup!

Coop reminded me that its done. And that it was done when I asked last time too. Adding this note so I dont stumble across comment#29 and worry anymore. :-)