778881 - (tegra-145) tegra-145 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Justin Wood (:Callek)]

Hitting twistd errors:

2012-08-06 20:07:49,343 DEBUG   MainProcess: Unhandled Error
2012-08-06 20:07:49,344 DEBUG   MainProcess: Traceback (most recent call last):
2012-08-06 20:07:49,344 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/application/app.py", line 622, in run
2012-08-06 20:07:49,344 DEBUG   MainProcess:     runApp(config)
2012-08-06 20:07:49,344 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/scripts/twistd.py", line 23, in runApp
2012-08-06 20:07:49,344 DEBUG   MainProcess:     _SomeApplicationRunner(config).run()
2012-08-06 20:07:49,344 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/application/app.py", line 376, in run
2012-08-06 20:07:49,344 DEBUG   MainProcess:     self.application = elf.createOrGetApplication()
2012-08-06 20:07:49,344 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/
lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/application/app.py", line 441, in createOrGetApplication
2012-08-06 20:07:49,345 DEBUG   MainProcess:     application = getApplication(self.config, passphrase)
2012-08-06 20:07:49,345 DEBUG   MainProcess: --- <exception caught here> ---
2012-08-06 20:07:49,345 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/application/app.py", line 452, in getApplication
2012-08-06 20:07:49,345 DEBUG   MainProcess:     application = service.loadApplication(filename, style, passphrase)
2012-08-06 20:07:49,345 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/application/service.py", line 400, in loadApplication
2012-08-06 20:07:49,345 DEBUG   MainProcess:     application = sob.loadValueFromFile(filename, 'application', passphrase)
2012-08-06 20:07:49,345 DEBUG   MainProcess:   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Twisted-11.0.0-py2.6-macosx-10.6-x86_64.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile
2012-08-06 20:07:49,345 DEBUG   MainProcess:     exec fileObj in d, d
2012-08-06 20:07:49,346 DEBUG   MainProcess: exceptions.SyntaxError: invalid syntax (buildbot.tac, line 11)
2012-08-06 20:07:49,346 DEBUG   MainProcess:
2012-08-06 20:07:49,346 DEBUG   MainProcess: Failed to load application: invalid syntax (buildbot.tac, line 11)
2012-08-06 20:07:49,346 DEBUG   MainProcess:
2012-08-06 20:07:49,346 INFO    MainProcess: buildslave start returned 1

Fixed with removing the |print TEST1| ... |print TEST5| stuff sprinkled in here, I don't remember doing that for this tegra at all, but came up fine.

Comment 2

[Carsten Book [:Tomcat]]

agent check is failing, trying a pdu reboot

Comment 3

[Carsten Book [:Tomcat]]

and back 

01/31/2013 06:45:39: INFO:  Tegra ID M    Tegra       CP    Slave :: Msg
2013-01-31 06:45:39,916  Tegra ID M    Tegra       CP    Slave :: Msg
01/31/2013 06:45:45: INFO: tegra-145 p    online   active   active :: 
2013-01-31 06:45:45,975 tegra-145 p    online   active   active :: 
bash-3.2$ ps auxw | grep "tegra-145"
cltbld   19868   0.0  0.1  2463868  11708   ??  S     6:42AM   0:00.97 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python /opt/local/bin/twistd --no_save --rundir=/builds/tegra-145 --pidfile=/builds/tegra-145/twistd.pid --python=/builds/tegra-145/buildbot.tac
cltbld   57247   0.0  0.0  2447208   2864   ??  S    10Dec12   6:02.48 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --device=tegra-145
cltbld   57246   0.0  0.0  2456948   3892   ??  S    10Dec12   1:33.17 /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python clientproxy.py -b --device=tegra-145
cltbld   19973   0.0  0.0  2435468    308 s001  U+    6:46AM   0:00.00 grep tegra-145

Comment 4

[Van Le [:van]]

SD card replaced, tegra reimaged and flashed.

[[e-mail]3 ~]$ fping tegra-145.tegra.releng.scl3.mozilla.com
tegra-145.tegra.releng.scl3.mozilla.com is alive

Comment 5

[Jordan Lund (:jlund)]

this machine is back in production and greening. closing for now

Comment 6

[Phil Ringnalda (:philor)]

Hasn't taken a job for 10 days.

Comment 7

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 8

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-145.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.109...
Connected to tegra-145.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 9

[Phil Ringnalda (:philor)]

Reenabled.