859196 - (foopy31) foopy31 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=21530989&tree=Graphics
https://tbpl.mozilla.org/php/getParsedLog.php?id=21530997&tree=Graphics
https://tbpl.mozilla.org/php/getParsedLog.php?id=21530927&tree=Graphics
https://tbpl.mozilla.org/php/getParsedLog.php?id=21531261&tree=Graphics

It seems to have a read-only filesystem.

Comment 1

[Justin Wood (:Callek)]

[cltbld@foopy31 ~]$ echo "hi" > tmp
-bash: tmp: Read-only file system

2013-04-07 20:06:28 -- Watcher is still alive
2013-04-07 20:06:29 -- ..checking tegra-320
2013-04-07 20:06:29 -- ..checking tegra-322
2013-04-07 20:06:29 -- ..checking tegra-327
2013-04-07 20:06:29 -- ..checking tegra-328
2013-04-07 20:06:29 -- ..checking tegra-329
2013-04-07 20:06:29 -- ..checking tegra-333
2013-04-07 20:06:29 -- ..checking tegra-335
2013-04-07 20:06:29 -- ..checking tegra-336
2013-04-07 20:06:29 -- ..checking tegra-337
2013-04-07 20:06:29 -- ..checking tegra-341
2013-04-07 20:06:29 -- ..checking tegra-344
2013-04-07 20:06:29 -- ..checking tegra-345
2013-04-07 20:06:29 -- ..checking tegra-348
2013-04-07 20:06:29 -- ..done next run in 300 seconds.
2013-04-07 20:11:29 -- Watcher is still alive
tee: /builds/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-320
/builds/watch_devices.sh: line 30: /builds/tegra-320/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-322
/builds/watch_devices.sh: line 30: /builds/tegra-322/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-327
/builds/watch_devices.sh: line 30: /builds/tegra-327/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-328
/builds/watch_devices.sh: line 30: /builds/tegra-328/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-329
/builds/watch_devices.sh: line 30: /builds/tegra-329/watcher.log: Read-only file system
2013-04-07 20:11:29 -- ..checking tegra-333
/builds/watch_devices.sh: line 30: /builds/tegra-333/watcher.log: Read-only file system

So it happened around that time, I couldn't find anything obvious to trigger it in my brief look.

Rebooted the foopy, did not turn on any devices from the reboot though.

Amy/Dustin can you take a peek for any understandable causes here.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

It's down now with unrecoverable disk errors.

I'll see if I get some disk diagnostics running.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Er, actually, DC Ops, can you take a look at that system?  It failed on boot.

Comment 4

[acoh]

I rebooted the server and based on the error message during boot the drive is failing. Is there anything you would like us to do with the server before we replace the drive?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Nope, have at it.

Comment 6

[Van Le [:van]]

RMA processed.

Case ID is 4644526102

Comment 7

[Van Le [:van]]

Drive should be in MV.

http://wwwapps.ups.com/etracking/tracking.cgi?TypeOfInquiryNumber=T&InquiryNumber1=1ZW6696X0121474104

Comment 8

[Van Le [:van]]

Drive replaced, please kickstart the host and let me know if there are any issues.

Comment 9

[Amy Rich [:arr] [:arich]]

Machine reimaged.

Comment 10

[Phil Ringnalda (:philor)]

<nagios-releng>: Sat 21:04:52 PDT [4164] foopy31.tegra.releng.scl3.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%

(every five minutes since sometime Friday night)

Comment 11

[Justin Wood (:Callek)]

I'm marking this foopy to be decomm'd, and decomming the attached tegras as well.

Those tegras are:

$ grep -B1 -A3 "foopy31" buildfarm/mobile/devices.json | grep "tegra-"
    "tegra-203": {
    "tegra-205": {
    "tegra-275": {
    "tegra-287": {
    "tegra-329": {
    "tegra-335": {
    "tegra-344": {
    "tegra-345": {
    "tegra-348": {

Comment 12

[Justin Wood (:Callek)]

done being decomm'd