546490 - some new linux ix machines hung at grub

Status: RESOLVED DUPLICATE of bug 546424

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Some of the Linux ix machines from bug 545134 are hung at a 'GRUB' screen. Power cycling doesn't help. Can someone look into this, please?

mv-moz2-linux-ix-slave{01,08} are both in this state, and possibly others.

Comment 1

[matthew zeier [:mrz]]

Dropping sev, no one's at the office right now to look.

What changed?  They were working when I handed them off and survived lots of reboots in the process.

Comment 2

[Chris AtLee [:catlee]]

One possibility is that we changed the SATA mode from IDE to AHCI.  The latter has about 50x better drive performance.  But they survived plenty of reboots after that too.

Comment 3

[Shyam Mani [:fox2mike]]

Ben, What steps did you undertake before the reboot? Where did you get the kernel from?

Comment 4

[matthew zeier [:mrz]]

I reverted slave01 back to IDE but same results.  At this point it'll have to sit until the morning when someone can boot off a rescue image.  

Sounds like GRUB is missing its stage2 image or /boot is gone/broken.  Worst case these will all need to be re-imaged and we don't have the master image onsite that I'm aware of.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> Ben, What steps did you undertake before the reboot? Where did you get the
> kernel from?


Also, bhearsum, are there any recent changes to puppet manifests that might have horked these machines all automatically?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> Ben, What steps did you undertake before the reboot? Where did you get the
> kernel from?

I don't know where the kernel came from; Catlee did that work.

> Also, bhearsum, are there any recent changes to puppet manifests that might
> have horked these machines all automatically?

If there's a Puppet change that broke things I'd be very surprised to find any of them working, but I'll certainly have a look.

Comment 7

[matthew zeier [:mrz]]

mrz notes

catlee
	slave01, changed SATA type
	rebooted okay, builds faster
	applied BIOS change across all Linux slaves
	
	Noticed not seeing all 4GB RAM,
		install PAE kernel
		slave01 - installed PAE
			yum install kernel-pae-i686
			rebooted via console
			manually selected PAE kernel
			no NIC driver
			slave01 was booting by default into PAE kernel with network
			
			
			slave8 & 25
				SATA mode changed
				PAE kernel installed, not default

Comment 8

[Jeremy Orem [:oremj]]

Assigning to mrz since it is paging.

Comment 9

[matthew zeier [:mrz]]

I believe catlee has this under control, punting over.

Comment 10

[Chris AtLee [:catlee]]

running grub-install /dev/sda seems to have fixed both slave01, and slave08. 
Still not sure what the initial cause was.

There's a rescue .iso image on the desktop of admin.b.m.o.  If you boot the
slave off of that, and then select 'grubdisk' from the first menu, then
"AUTOMAGIC BOOT", and then select the 2.6.18 kernel to boot from, you can then
run 'grub-install /dev/sda' as root.