730736 - (tegra-056) tegra-056 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

This is more recent bustage than most of my "every single run in the last 300 hasn't even gotten as far as being orange or red" ones, but no less real. Of its last 60 runs, one was red, 28 were purple, and 31 were retry.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

stop_cp has been run against this tegra.

Comment 2

[Matthew Larrain[:MaRu]]

reimaged

Comment 3

[Mike Taylor [:bear]]

cp restarted

Comment 4

[Justin Wood (:Callek)]

Last job 9 days, 9:18:26 ago

telnet is not giving a SUTAgent prompt, but connects just fine, issued a PDU cycle.

Comment 5

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15942795&tree=Mozilla-Inbound

Comment 6

[Justin Wood (:Callek)]

(In reply to Phil Ringnalda (:philor) from comment #5)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=15942795&tree=Mozilla-
> Inbound

As far as I can tell this failure was not a bad-device failure, but we're still investigating among other code issues that got deployed within the past 2 weeks.

Connected, with green jobs through its last 10 runs or so.

Comment 7

[Justin Wood (:Callek)]

The device had never come up after a reboot around Nov 27 16:12 -- I just manually powercycled and stop/start cp cycle. It should come back up fine.

Comment 8

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #7)
> It should come back up fine.

...And it did not. Reimage please

Comment 9

[Justin Wood (:Callek)]

cp was down for this device, ran start_cp --> will check in later

Comment 10

[Justin Wood (:Callek)]

now taking jobs

Comment 11

[Justin Wood (:Callek)]

Was reimaged today, but is still failing:

tegra-056 -  not running -    enabled - 'Remote Device Error: unable to write to sdcard'

Comment 12

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 13

[Trey Franco [:tfranco]]

 
PING tegra-056.build.mtv1.mozilla.com (10.250.49.43): 56 data bytes
64 bytes from 10.250.49.43: icmp_seq=0 ttl=63 time=42.698 ms
64 bytes from 10.250.49.43: icmp_seq=1 ttl=63 time=7.447 ms
64 bytes from 10.250.49.43: icmp_seq=2 ttl=63 time=5.143 ms

PING tegra-078.build.mtv1.mozilla.com (10.250.49.66): 56 data bytes
64 bytes from 10.250.49.66: icmp_seq=0 ttl=63 time=5.100 ms
64 bytes from 10.250.49.66: icmp_seq=1 ttl=63 time=4.892 ms
64 bytes from 10.250.49.66: icmp_seq=2 ttl=63 time=2.820 ms

PING tegra-247.build.mtv1.mozilla.com (10.250.51.87): 56 data bytes
64 bytes from 10.250.51.87: icmp_seq=0 ttl=63 time=16.129 ms
64 bytes from 10.250.51.87: icmp_seq=1 ttl=63 time=29.851 ms
64 bytes from 10.250.51.87: icmp_seq=2 ttl=63 time=27.273 ms

PING tegra-195.build.mtv1.mozilla.com (10.250.50.105): 56 data bytes
64 bytes from 10.250.50.105: icmp_seq=0 ttl=63 time=5.197 ms
64 bytes from 10.250.50.105: icmp_seq=1 ttl=63 time=5.311 ms
64 bytes from 10.250.50.105: icmp_seq=2 ttl=63 time=5.136 ms

PING tegra-096.build.mtv1.mozilla.com (10.250.49.84): 56 data bytes
64 bytes from 10.250.49.84: icmp_seq=0 ttl=63 time=15.703 ms
64 bytes from 10.250.49.84: icmp_seq=1 ttl=63 time=2.565 ms 

tegra-088.build.mtv1.mozilla.com (10.250.49.76): 56 data bytes
64 bytes from 10.250.49.76: icmp_seq=0 ttl=63 time=5.074 ms
64 bytes from 10.250.49.76: icmp_seq=1 ttl=63 time=5.221 ms
64 bytes from 10.250.49.76: icmp_seq=2 ttl=63 time=5.935 ms

tegra-072.build.mtv1.mozilla.com (10.250.49.59): 56 data bytes
64 bytes from 10.250.49.59: icmp_seq=0 ttl=63 time=5.121 ms
64 bytes from 10.250.49.59: icmp_seq=1 ttl=63 time=8.175 ms
64 bytes from 10.250.49.59: icmp_seq=2 ttl=63 time=20.454 ms

tegra-093.build.mtv1.mozilla.com (10.250.49.81): 56 data bytes
64 bytes from 10.250.49.81: icmp_seq=0 ttl=63 time=26.527 ms
64 bytes from 10.250.49.81: icmp_seq=1 ttl=63 time=6.334 ms
64 bytes from 10.250.49.81: icmp_seq=2 ttl=63 time=2.242 ms

tegra-038.build.mtv1.mozilla.com (10.250.49.25): 56 data bytes
64 bytes from 10.250.49.25: icmp_seq=0 ttl=63 time=10.674 ms
64 bytes from 10.250.49.25: icmp_seq=1 ttl=63 time=5.123 ms
64 bytes from 10.250.49.25: icmp_seq=2 ttl=63 time=2.959 ms

Comment 14

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 15

[Van Le [:van]]

flashed and reimaged

Comment 16

[Justin Wood (:Callek)]

we just can't get this one to stay up, --> decom

Comment 17

[[e-mail] (:bhearsum)]

decommed