778814 - (tegra-040) tegra-040 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Justin Wood (:Callek)]

Last job 31 days, 6:46:38 ago

Had error.flg [Remote Device Error: Unable to properly remove /mnt/sdcard/tests] 

Did a remote sdcard format -- todo -- look back in on it.

Comment 2

[Justin Wood (:Callek)]

that fixed it...

Comment 3

[[e-mail] (:bhearsum)]

Throwing error.flg [Remote Device Error: Unable to properly remove /mnt/sdcard/tests]  again. I'm trying a remote format.

Comment 4

[[e-mail] (:bhearsum)]

Back in production....for now.

Comment 5

[Kim Moir [:kmoir] ET]

Manually reformatted its sdcard today, pdu rebooted, and cleared the error flag. It's back in production and running builds.

Comment 6

[Justin Wood (:Callek)]

was cycling with 2012-12-04 04:12:01,092 INFO    MainProcess: buildslave start returned 1 again.
After a connection-lost to bm20. I ./stop_cp and ./start_cp cycled it.

Comment 7

[Justin Wood (:Callek)]

Back to being dead, please swap SDCard and reimage.

Comment 8

[Carsten Book [:Tomcat]]

needed to be rebooted -> checking that it comes up again

Comment 9

[Carsten Book [:Tomcat]]

and we have it back 

01/31/2013 05:01:21: INFO:  Tegra ID M    Tegra       CP    Slave :: Msg
2013-01-31 05:01:21,464  Tegra ID M    Tegra       CP    Slave :: Msg
01/31/2013 05:01:27: INFO: tegra-040 p    online   active   active :: 
2013-01-31 05:01:27,566 tegra-040 p    online   active   active ::

Comment 10

[Carsten Book [:Tomcat]]

also rebooted again and its back

bash-3.2$ ps auxw | grep "tegra-040"
cltbld   24367   0.0  0.1  2447536   2828   ??  S     3:50AM   0:00.03 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --device=tegra-040
cltbld   24366   0.0  0.1  2457276   3332   ??  S     3:50AM   0:00.02 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --device=tegra-040
cltbld   24492   0.0  0.3  2464032  11624   ??  S     3:54AM   0:01.30 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/twistd --no_save --rundir=/builds/tegra-040 --pidfile=/builds/tegra-040/twistd.pid --python=/builds/tegra-040/buildbot.tac
bash-3.2$ ./check.sh -t tegra-040 
02/01/2013 03:54:58: INFO:  Tegra ID M    Tegra       CP    Slave :: Msg
2013-02-01 03:54:58,725  Tegra ID M    Tegra       CP    Slave :: Msg
02/01/2013 03:55:04: INFO: tegra-040 p    online   active   active :: 
2013-02-01 03:55:04,771 tegra-040 p    online   active   active :: 
bash-3.2$

Comment 11

[Justin Wood (:Callek)]

Monday, September 30, 2013 7:37:21 PM

Comment 12

[Van Le [:van]]

replaced SD and flashed/reimaged, however it seems this tegra is overheating and there's a slight burning smell. if it fails again, we should decommission it.

Comment 13

[Eric Ramirez [:Eric]]

sdcard replaced and reimaged/flashed.

Comment 14

[Justin Wood (:Callek)]

handled in last recovery on Dec 16.

Comment 15

[[e-mail] (:bhearsum)]

2014-01-16 10:10:16 tegra-040 p   OFFLINE   active  OFFLINE :: PING tegra-040.tegra.releng.scl3.mozilla.com (10.26.85.27) 56(84) bytes of data. From foopy112.tegra.releng.scl3.mozilla.com (10.26.84.12) icmp_seq=2 Destination Host Unreachable;error.flg [Automation Error: Unable to ping device after 5 attempts] 


pdu reboot didn't help

Comment 16

[Eric Ramirez [:Eric]]

SD card reimaged/flashed.

Comment 17

[Justin Wood (:Callek)]

Still failing to mount an sdcard

Comment 18

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 966371)

Comment 19

[Chris Cooper [:coop] (he/him)]

error.flg is set to:

Automation Error: Unable to connect to device after 5 attempts

Comment 20

[Van Le [:van]]

SD card flashed and tegra reimaged. if it fails again we can replace the SD card.

[[e-mail]3 ~]$ telnet tegra-040.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.27...
Connected to tegra-040.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>

Comment 21

[Justin Wood (:Callek)]

Taking jobs again