781897 - (tegra-113) tegra-113 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Justin Wood (:Callek)]

tegra-113 has failed the last lot of jobs (at least the last 10)

Running stop_cp on it, a PDU powercycle, and then verify, but lets recover it to be extra safe.

Comment 1

[Justin Wood (:Callek)]

[old bug by quite a while -- forgot to set whiteboard so got lost in the tangle, need to verify its still an issue before we tie it back to another recovery bug]

Comment 2

[[e-mail] (:bhearsum)]

Back in production.

Comment 3

[Phil Ringnalda (:philor)]

https://secure.pub.build.mozilla.org/buildapi/recent/tegra-113 - it isn't actually what I'd call better.

Comment 4

[Phil Ringnalda (:philor)]

13 in a row.

Comment 5

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15453011&tree=Profiling

Comment 6

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15452588&tree=Try

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15451526&tree=Try

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15450029&tree=Try

Comment 9

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15447830&tree=Try

Comment 10

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15444067&tree=Try

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

stop_cp run on foopy13.

Comment 12

[Phil Ringnalda (:philor)]

No real skin off my nose anymore, but it shambled back to the land of the living, did a few green and orange runs, and has settled into setting RETRY or being purple for the last 65 or so.

Comment 13

[Justin Wood (:Callek)]

[TO DO -- recover]

needs a proper recovery, maybe this time Dr. Frankenstein will get it right?

(if he doesn't we'll bury it, ring its grave in a moat of holy water, stuff the cofin with garlic, and leave a silver bullet and a wooden stake both through its heart, and hang a crucifix above its grave site.)

Ensured stop_cp run for now, pending its visit to Frankenstein's lab

Comment 14

[[e-mail] (:bhearsum)]

Back in production. For how long? Who knows...

Comment 15

[Phil Ringnalda (:philor)]

Stinking up the place something fierce, giving tegras a bad name, 28% green.

Comment 16

[Justin Wood (:Callek)]

Ran ./stop_cp.sh

Comment 17

[Justin Wood (:Callek)]

Brought bake to life.

Comment 18

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #17)
> Brought bake to life.

On second thought, this was just a muscle spasm -- ran stop_cp and sending to recovery, please swap sdcard and reimage.

If we had a biopsy device I'd confirm the cancer or not.

Comment 19

[Ashlee Nguyen [:ashlee]]

Tegra-113 reimaged + sd card swap, pingable

--- tegra-113.build.mtv1.mozilla.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss

Comment 20

[Justin Wood (:Callek)]

Thursday, August 29, 2013 4:07:39 PM

Comment 21

[Salvador Espinoza [:sal]]

flashed and reimaged

Comment 22

[[e-mail] (:bhearsum)]

Only took one job before going down. Callek?

Comment 23

[Justin Wood (:Callek)]

handled in last recovery on Dec 16.

Comment 24

[Van Le [:van]]

SD card formatted, tegra reimaged and flashed.

[[e-mail]3 ~]$ fping tegra-113.tegra.releng.scl3.mozilla.com
tegra-113.tegra.releng.scl3.mozilla.com is alive

Comment 25

[Phil Ringnalda (:philor)]

Hasn't taken a job for 4 days.

Comment 26

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 27

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-113.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.88...
Connected to tegra-113.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 28

[Phil Ringnalda (:philor)]

Reenabled.

Comment 29

[Justin Wood (:Callek)]

foopy117 is going away, and these tegras are attached to it.

They are being decomissioned in Bug 1043938, and I just disabled all of them in slavealloc and marked them as decom with a comment pointing at Bug 1043938