736911 - (bld-linux64-ix-004) bld-linux64-ix-004 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

It's down and I can't get to the ipmi interface either.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

It's back now without intervention, timeline:

16:28 - twistd.log - finished an android-xul job on try, initiated reboot
16:29 - twistd.log - back on bm14, sits idle
16:37 - #buildduty - all the nagios checks timeout after 15 or 30 seconds
18:41 - twistd.log - loses master, tries to reconnect but has DNS failures until ...
19:41 - twistd.log - idleizer reboots it
19:42 - #buildduty - 88% packet loss (instead of 100)
19:47 - #buildduty - all checks go green
19:53 - twistd.log - starts android job

No history of issues in the list of recent builds.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Has a hard disk failure, screenshot to follow. It doesn't boot with the following error on POST:

SATA Port 0 ST3250318AS  CC38
       S.M.A.R.T Capable and Status BAD
....
AHCI Port0 Device Error
Press F1 to Resume
----

Interestingly it does boot if you use F1 but seems like a bad idea to trust it, even on try.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Repuppeted and back in the try pool.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

This became bld-linux64-ix-004 in bug 847529.

It has a problem with /:

Checking filesystems
/dev/sda3 contains a file system with errors, check forced.
Error reading block 42691483 (Attempt to read block from filesystem resulted in short read) while reading directory block.

/dev/sda3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: screenshot

Running fsck yields quite a few "Buffer I/O error on device sda3", which seems to mean the disk is packing up.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

arr, would you recommend reimage or replace disk ?

Comment 7

[Amy Rich [:arr] [:arich]]

nthomas: please open up a bug with dcops and have them investigate/run some diags.  This machine is still under warranty till the beginning of June, so it's a hardware problem, let's get it taken care of now.

Comment 8

[Chris AtLee [:catlee]]

If this doesn't get reimaged, we need to copy over the correct xrbld ssh keys before putting this back into production. (https://bugzilla.mozilla.org/show_bug.cgi?id=837234#c23)

Comment 9

[Justin Wood (:Callek)]

Enabled in slavealloc after image+new disk and copied ssh keys:

[cltbld@bld-linux64-ix-004 ~]$ scp cltbld@bld-linux64-ix-015:.ssh/* .ssh/

And rebooted

Comment 10

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #9)
> Enabled in slavealloc ...

Re-disabled. This host was pulled from the buildbot slave list when we reimaged many around this number as w64, did we intend to NOT make this a windows host, or what?

Comment 11

[Armen [:armenzg]]

I think I didn't ask for it to be re-imaged as win64 because it had a note on slavealloc (maybe bad drive) that made me worry and pick another machine instead.

Comment 12

[Justin Wood (:Callek)]

Attachment: [configs] add back to slaves list

Comment 13

[Armen [:armenzg]]

Comment on attachment 741824 [details] [diff] [review]
[configs] add back to slaves list

Review of attachment 741824 [details] [diff] [review]:
-----------------------------------------------------------------

Oh... I even removed it! Sorry :(

Comment 14

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/buildbot-configs/rev/70a0b41822a9

Comment 15

[Justin Wood (:Callek)]

change in production (todo put machine back in pool)

Comment 16

[Justin Wood (:Callek)]

re-enabled and rebooted.