734798 - (tegra-036) tegra-036 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

According to my counting on my fingers stats, it has failed 30% of its jobs over the last 200 and 40% of its jobs over the last 50, which puts it above my usual threshold of "this one is failing 50%, it's unquestionably broken" but solidly at the bottom of the "this is one crappy tegra" range. So, I'd like to be able to watch what it does post-reimaging, to see whether the bottom of that range can actually be pulled up into the horrible normal of around 20% failures.

Comment 1

[Aki Sasaki (not active)]

Ran stop_cp.sh.

Comment 2

[Chris Cooper [:coop] (he/him)]

Already back in production.

Comment 3

[Phil Ringnalda (:philor)]

Failed every one of the last 25 jobs.

Comment 4

[Justin Wood (:Callek)]

stop_cp run on it

Comment 5

[[e-mail] (:bhearsum)]

Back in production.

Comment 6

[Phil Ringnalda (:philor)]

https://secure.pub.build.mozilla.org/buildapi/recent/tegra-036 - that didn't help.

Comment 7

[Phil Ringnalda (:philor)]

8 red in a row.

Comment 8

[[e-mail] (:bhearsum)]

Ran stop_cp.sh again.

Comment 9

[Phil Ringnalda (:philor)]

SUTAgent deploy brought it back out of the grave, dripping clots of dirt and flesh. Three green runs, but it'll turn back to the red side and I'll be back.

Comment 10

[Phil Ringnalda (:philor)]

And, no surprise, I'm back: 4 reds in a row, and no reason to think it'll turn back toward the light.

Comment 11

[Phil Ringnalda (:philor)]

s/4 reds in a row/10 reds in a row/

Too bad the ability to correctly predict whether a tegra has hit a brief bad stretch, or has turned into pure blood-dripping evil, isn't more broadly useful.

Comment 12

[Justin Wood (:Callek)]

I've now sent Milla Jovovich after this Zombie Device, and she did her dastardliest. (she found the stop_cp.sh command and ran it)

Comment 13

[Justin Wood (:Callek)]

rumor has it that DCOps learned some new magic tricks at hogwarts recently, or maybe they are the same tricks with just more flair. Either way, lets see if this tegra falls for the tricks, or just tries to eat more brains. -- Assigning to recovery one last time (if this fails within a week of recovery, lets kill it with thermite)

Comment 14

[Justin Wood (:Callek)]

So they reimaged, start_cp running now for it

Comment 15

[Phil Ringnalda (:philor)]

49 straight retries.

Comment 16

[Phil Ringnalda (:philor)]

After an impressive run of 54 retires in a row, it tripped over its own feet, did a purple and check itself offline.

Comment 17

[Ed Morley [:emorley]]

3 passes out of 25 runs:
https://secure.pub.build.mozilla.org/buildapi/recent/tegra-036

Please can this tegra be disabled until investigated/reimaged/beaten with a bat.

Comment 18

[Rail Aliiev [:rail]]

disabled in slavealloc

Comment 19

[Rail Aliiev [:rail]]

err, stopped.

Comment 20

[Justin Wood (:Callek)]

(In reply to Rail Aliiev [:rail] from comment #19)
> err, stopped.

*actually* stopped now -- apparantly it wasn't

bash-3.2$ ps auxwww | grep tegra-036
cltbld   28349   0.0  0.1  2447024   2824   ??  S    23Oct12   4:53.19 /opt/local/Library/Frameworks/Python.frame
work/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --tegra=tegra-036 --debug
cltbld   28348   0.0  0.1  2457020   3808   ??  S    23Oct12   1:39.39 /opt/local/Library/Frameworks/Python.frame
work/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --tegra=tegra-036 --debug
cltbld   59447   0.0  0.0  2425520    116 s002  R+    3:42AM   0:00.00 grep tegra-036

Comment 21

[Justin Wood (:Callek)]

No jobs taken on this device for >= 7 weeks

Comment 22

[Justin Wood (:Callek)]

(mass change: filter on tegraCallek02reboot2013)

I just rebooted this device, hoping that many of the ones I'm doing tonight come back automatically. I'll check back in tomorrow to see if it did, if it does not I'll triage next step manually on a per-device basis.

---
Command I used (with a manual patch to the fabric script to allow this command)

(fabric)[jwood@dev-master01 fabric]$  python manage_foopies.py -j15 -f devices.json `for i in 021 032 036 039 046  048 061 064 066 067 071 074 079 081 082 083 084 088 093 104 106 108 115 116 118 129 152 154 164 168 169 174 179 182 184 187 189 200 207 217 223 228 234 248 255 264 270 277 285 290 294 295 297 298 300 302 304 305 306 307 308 309 310 311 312 314 315 316 319 320 321 322 323 324 325 326 328 329 330 331 332 333 335 336 337 338 339 340 341 342 343 345 346 347 348 349 350 354 355 356 358 359 360 361 362 363 364 365 367 368 369; do echo '-D' tegra-$i; done` reboot_tegra

The command does the reboot, one-at-a-time from the foopy the device is connected from. with one ssh connection per foopy

Comment 23

[Justin Wood (:Callek)]

Buildbot is properly connected and happy now.

Comment 24

[Justin Wood (:Callek)]

Wednesday, October 02, 2013 4:21:17 AM

Comment 25

[Van Le [:van]]

flashed and reimaged. if it fails again, we should replace the SD card.

Comment 26

[[e-mail] (:bhearsum)]

Back in production

Comment 27

[Van Le [:van]]

SD card replaced, tegra reimaged and flashed.

[[e-mail]3 ~]$ fping tegra-036.tegra.releng.scl3.mozilla.com
tegra-036.tegra.releng.scl3.mozilla.com is alive

Comment 28

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 29

[Phil Ringnalda (:philor)]

Hasn't taken a job for 13 days.

Comment 30

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 31

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-036.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.23...
Connected to tegra-036.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 32

[Phil Ringnalda (:philor)]

Reenabled.