854597 - kickstarting talos-linux64-ix-057 is very slow

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Amy Rich [:arr] [:arich]]

Trying to kickstart this host takes a very long time, pointing to some sort of hardware issue.  We've already tried replacing the cable and changing switch ports without effect.  I asked hubear to run diagnostics on it to see if we can figure out what the issue is (network, disk, RAM, etc).

Comment 1

[Ashlee Nguyen [:ashlee]]

I went ahead and filed a ticket for iX Systems and described the problem above. As soon as I get a reply, I'll update the bug. They're pretty good at responding quickly.

Ticket ID: DHG-111212

Comment 2

[Ashlee Nguyen [:ashlee]]

iX Systems suggested us trying this before we request for a replacement. 

Remove the node for at least 20 seconds and place it back into the chassis.

If you continue to experience issues afterward, please let us know and we will promptly schedule a pick-up (for repair/replacement).

Comment 3

[Hubert Eulalia [:Hubear]]

We reseated it a couple of times already, we had no luck with it.

Comment 4

[Van Le [:van]]

Host is back from iX systems with it's motherboard replaced. nic0 and mgmt mac addresses have been updated in inventory. Please kickstart host and close bug if issue is resolved.

Comment 5

[Amy Rich [:arr] [:arich]]

This machine still doesn't seem happy.  I wonder if it's a bad disk, too?  Or maybe something that's not seated properly?

Comment 6

[Van Le [:van]]

:hubear, can you grab a drive from one of the new 10 iX systems that came in and swap it with this host? Set the suspect drive aside so we can RMA it if necessary.

Comment 7

[Hubert Eulalia [:Hubear]]

Hard drive swapped, Please kickstart host and close bug if issue is resolved.

Comment 8

[Amy Rich [:arr] [:arich]]

Now it doesn't even PXE boot.  All I get after the initial power on screen is a blinking cursor.

Comment 9

[Ashlee Nguyen [:ashlee]]

I filed a ticket for iXsystems and let them know its still acting up.

Comment 10

[Ashlee Nguyen [:ashlee]]

:arr iX Systems suggested we try this: 
Blank screen usually means there is an invalid boot block on the boot device. Please try booting the system and while the BIOS screens are up press F12 repeatedly. This will instruct BIOS to PXE boot instead of following the standard boot path.

Comment 11

[Derek Moore [:dmoore]]

:ashlee, :hubear

Can you two put a monitor on this host and perform some onsite debugging? We want to try reseating all the components and confirming we can actually reach the PXE boot screen before returning it to :arr. She can't do anything remotely until the host can reach that screen.

Comment 12

[Hubert Eulalia [:Hubear]]

:dmoore

The host is now on the PXE boot screen.

Comment 13

[Amy Rich [:arr] [:arich]]

Still slow.

Comment 14

[Ashlee Nguyen [:ashlee]]

We'll try moving it to one of the chassis we received this week in order to troubleshoot the problem.

Comment 15

[Van Le [:van]]

:arr, I swapped the location of talos-linux64-ix-057.test.releng.scl3.mozilla.com and talos-linux32-ix-100.test.releng.scl3.mozilla.com. Can you kickstart both and let me know of any issues?

Comment 16

[Amy Rich [:arr] [:arich]]

This seems to be functioning normally now after some reseating.