778838 - (tegra-141) tegra-141 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Justin Wood (:Callek)]

Manual PDU cycle got this device working again.

Comment 2

[Phil Ringnalda (:philor)]

17 reds in a row, https://secure.pub.build.mozilla.org/buildapi/recent/tegra-141

Comment 3

[Justin Wood (:Callek)]

ran stop_cp on it, and vin says DCOps will grab it right now, since they are still in the colo

Comment 4

[[e-mail] (:bhearsum)]

Back in production.

Comment 5

[Phil Ringnalda (:philor)]

Has a sort of odd failure pattern, according to https://secure.pub.build.mozilla.org/buildapi/recent/tegra-141?numbuilds=50, but it also has a completely unacceptable failure rate.

Comment 6

[Phil Ringnalda (:philor)]

12 red in row, not counting one killed on try.

Comment 7

[Phil Ringnalda (:philor)]

14.

Comment 8

[Phil Ringnalda (:philor)]

16.

Comment 9

[Justin Wood (:Callek)]

ran stop_cp on it

Comment 10

[Justin Wood (:Callek)]

IT handled this, start_cp run now

Comment 11

[Ed Morley [:emorley]]

Fails every run; please re-disable.

Comment 12

[Ashlee Nguyen [:ashlee]]

Tegra-141 reimaged and pingable + swapped sd card

--- tegra-141.build.mtv1.mozilla.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss

Comment 13

[Phil Ringnalda (:philor)]

Its last 217 jobs in a row have all done nothing but set RETRY.

Comment 14

[Phil Ringnalda (:philor)]

Hi! It's three weeks later, and by not disabling this tegra over the last three weeks, we've allowed it to take 365 jobs, and set RETRY on 365 jobs. Counting the time to take the job and the time for another slave to actually take it and do it, say 7 minutes is a conservative estimate. This slave has slowed us down by 42.5 hours over the last three weeks.

Please disable it.

Comment 15

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from comment #14)
> Hi! It's three weeks later, and by not disabling this tegra over the last
> three weeks, we've allowed it to take 365 jobs, and set RETRY on 365 jobs.
> Counting the time to take the job and the time for another slave to actually
> take it and do it, say 7 minutes is a conservative estimate. This slave has
> slowed us down by 42.5 hours over the last three weeks.
> 
> Please disable it.

Comment 16

[Kim Moir [:kmoir] ET]

disabled

Comment 17

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 18

[Justin Wood (:Callek)]

Successfully recovered.

Comment 19

[Justin Wood (:Callek)]

Sunday, October 20, 2013 10:39:07 PM

Comment 20

[Salvador Espinoza [:sal]]

flashed and reimaged

Comment 21

[[e-mail] (:bhearsum)]

Back in production

Comment 22

[Van Le [:van]]

SD card replaced, tegra reimaged and flashed.

[[e-mail]3 ~]$ fping tegra-141.tegra.releng.scl3.mozilla.com
tegra-141.tegra.releng.scl3.mozilla.com is alive

Comment 23

[John Hopkins (:jhopkins)]

Power cycled, waited a day.

error.flg: Automation Error: Unable to connect to device after 5 attempts

Pings but can't connect via telnet.

Comment 24

[Jordan Lund (:jlund)]

in production running green

it had one retry:
Remote Device Error: Unable to copy file '/builds/tegra-141/talos-data/talos/page_load_test/tart/dbghelp.dll' to directory 'dbghelp.dll' on the remote device
    mozdevice.devicemanager.DMError: Automation Error: Error processing command 'mkdr /dbghelp.dll'; err='Could not create the directory /dbghelp.dll'

but this looked like an anomaly

Comment 25

[Phil Ringnalda (:philor)]

Hasn't taken a job for two weeks.

Comment 26

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 27

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-141.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.106...
Connected to tegra-141.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 28

[Phil Ringnalda (:philor)]

Reenabled.