601123 - please run hardware diagnostics on linux-ix-slave17

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[[e-mail] (:bhearsum)]

This machine repeatedly got into uninterruptible sleep while doing disk-heavy operations and required rebooting.

Comment 1

[Justin Lazaro [:jlaz] (use needinfo)]

Sent an email to IX support to take a look at this/ask for advice

Comment 2

[Justin Lazaro [:jlaz] (use needinfo)]

Just handed this machine (Asset tag #4636) to Ramsey of IX Systems, root password reset, so that they can debug this issue further.

Comment 3

[Justin Lazaro [:jlaz] (use needinfo)]

Looks like theyve found a bad disk on this machine, will update as soon as i find out more

Comment 4

[Justin Lazaro [:jlaz] (use needinfo)]

Got this back yesterday, here's an update from Matt Finney of IX regarding this machine:

--
Hello,

The system has been repaired. Originally the drive was disconnecting and reconnecting to SATA repeatedly while idle. I ran Seagate diagnostics on the drive, which found 2 bad sectors that I repaired. I replaced the SATA cable as well. 

I'm not able to reproduce any disk issues after those changes. 

--

Will bring back on next trip to Internap

Comment 5

[[e-mail] (:bhearsum)]

Thanks!

I guess the overall conclusion with these machines is that we'll treat issues on a case-by-case basis? Eg, IX Systems has not found anything that leads them to believe there's a larger issue at work?

Comment 6

[Justin Dow [:jabba]]

Brought machine to Internap and it is online now.

Note: This machine was originally at Castro, so RelEng will need to update it to work at Internap.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Please put the root password back to the RelEng value.

Comment 8

[Justin Dow [:jabba]]

password reset to current RelEng value.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I'm not sure this is fixed. It was pretty slow at cleaning up files from disk, so I started a fsck on /builds and 20 minutes later it's still on Pass 1. I'll let it run a bit more ...

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Took 35 minutes on a fairly empty 168G partition, which seems a bit slow. More info on bug 611128.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Yeah, definitely still some disk error on this, see bug 611128 comment #4 for details on a 12x slowdown. We could try re-imaging the machine if it's a data error, or do more diagnostics to look for a busted drive. Thoughts ?

Comment 12

[Justin Dow [:jabba]]

Is this out of production, such that I can re-image at will?

Comment 13

[Justin Dow [:jabba]]

ping?

Comment 14

[Chris AtLee [:catlee]]

I believe it's down.  I can't ssh to it in any case, and it hasn't done any work since october.

Comment 15

[Justin Dow [:jabba]]

I re-imaged it. It is currently stuck on trying to start puppet. Not sure how to get around that. Please check it out and see if things are better or not. The machine is reachable over IPMI at 10.12.48.253.

Comment 16

[Mike Taylor [:bear]]

grabbing the bug to look at it's puppet config and see if we can get it back online monday

Comment 17

[Corey Shields [:cshields]]

What is the status on this?  If the box is back up can we close this bug?

Comment 18

[Chris Cooper [:coop] (he/him)]

Filed bug 636827 for the post-imaging work by releng.