813513 - (tegra-348) tegra-348 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ed Morley [:emorley]]

40-50% failure rate:
https://secure.pub.build.mozilla.org/buildapi/recent/tegra-348

Please can we disable until investigated/reimaged/...

Comment 1

[Rail Aliiev [:rail]]

Stopped.

Comment 2

[[e-mail] (:bhearsum)]

pdu reboot
start_cp

Comment 3

[[e-mail] (:bhearsum)]

Still not back in production, back to recovery.

Comment 4

[Justin Wood (:Callek)]

(mass change: filter on tegraCallek02reboot2013)

I just rebooted this device, hoping that many of the ones I'm doing tonight come back automatically. I'll check back in tomorrow to see if it did, if it does not I'll triage next step manually on a per-device basis.

---
Command I used (with a manual patch to the fabric script to allow this command)

(fabric)[jwood@dev-master01 fabric]$  python manage_foopies.py -j15 -f devices.json `for i in 021 032 036 039 046  048 061 064 066 067 071 074 079 081 082 083 084 088 093 104 106 108 115 116 118 129 152 154 164 168 169 174 179 182 184 187 189 200 207 217 223 228 234 248 255 264 270 277 285 290 294 295 297 298 300 302 304 305 306 307 308 309 310 311 312 314 315 316 319 320 321 322 323 324 325 326 328 329 330 331 332 333 335 336 337 338 339 340 341 342 343 345 346 347 348 349 350 354 355 356 358 359 360 361 362 363 364 365 367 368 369; do echo '-D' tegra-$i; done` reboot_tegra

The command does the reboot, one-at-a-time from the foopy the device is connected from. with one ssh connection per foopy

Comment 5

[Justin Wood (:Callek)]

Back from recovery

Comment 6

[[e-mail] (:bhearsum)]

7 days, 0:01:08 since last job

Comment 7

[[e-mail] (:bhearsum)]

back in production

Comment 8

[Justin Wood (:Callek)]

last job was april 8, devices.json shows it as on foopy31, foopy31 doesn't. And its not listed in tegra-dashboard right now... may have to hunt for its real home or give it one.

Comment 9

[Justin Wood (:Callek)]

it doesn't have a real-home atm, added to foopy31, and is now up with buildbot

Comment 10

[Van Le [:van]]

flashed and reimaged

Comment 11

[[e-mail] (:bhearsum)]

This slave didn't stay up for long, found this in watcher.log:
649 2013-12-12 07:00:01 -- #################################################################
650 2013-12-12 07:00:01 -- # Starting cycle for our device (tegra-348 = 10.250.51.188) now #
651 2013-12-12 07:00:01 -- #################################################################
652 2013-12-12 07:00:01 -- contacting slavealloc
653 2013-12-12 07:00:01 -- Buildbot is not running
654 retry: Calling <function run_with_timeout at 0x7f3e56761488> with args: (['wget', '-q', '-O/builds/tegra-348/buildbot.tac.new', 'http://slavealloc.pvt.build.mozilla.o    rg/gettac/tegra-348'], 300, None, 'ERROR 404: Not Found', True, True), kwargs: {}, attempt #1
655 Executing: ['wget', '-q', '-O/builds/tegra-348/buildbot.tac.new', 'http://slavealloc.pvt.build.mozilla.org/gettac/tegra-348']
656 Process stderr:
657 
658 2013-12-12 07:00:03 -- error.flg file detected


Callek?

Comment 12

[[e-mail] (:bhearsum)]

Looks like it fixed itself

Comment 13

[Justin Wood (:Callek)]

Being decomm'd along with foopy31

Comment 14

[Justin Wood (:Callek)]

Decomm'd along with foopy31 in Bug 859196