722063 - (tegra-103) tegra-103 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

Looks like it broke on the morning of November 21st, since which time it has had 970 exception and retry runs in a row.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Ran stop_cp, clientproxy and buildslave are offline.

Comment 2

[Phil Ringnalda (:philor)]

Looks like someone reenabled it (or it can zombie) on February 13th, on which lucky day it did four green runs and then four red/purple. Then it got reenabled this morning around 10, did 7 green runs, then timed out running tests in https://tbpl.mozilla.org/php/getParsedLog.php?id=9834484&tree=Try (a typical bad-tegra thing to do, I'm starting to realize), then timed out in cleanup device in https://tbpl.mozilla.org/php/getParsedLog.php?id=9835666&tree=Mozilla-Inbound and if it stays true to form, will continue to fail now until it's disabled and reimaged (or disabled and put in staging, or disabled and thrown out back, not sure what happens after the part I like, "disabled").

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

'stop_cp tegra-103' has been run on foopy12.

Comment 4

[[e-mail] (:bhearsum)]

Back in production.

Comment 5

[Justin Wood (:Callek)]

dead again... PDU cycles are not even bringing up the telnet port we need. Please reimage.

Comment 6

[Justin Wood (:Callek)]

Apparantly it was just taking extra long to come up, working fine now.

Comment 7

[Justin Wood (:Callek)]

Last job 9 days, 19:47:52 ago

Could connect via telnet, no prompt

Also had a stale proc:
cltbld   32540   0.0  0.1  2456664  10468   ??  S    28Sep12   0:00.58 /opt/local/Library/Frameworks/Python.frame
work/Versions/2.6/Resources/Python.app/Contents/MacOS/Python /builds/tegra-103/talos-data/talos/bcontroller.py --
configFile /builds/tegra-103/talos-data/talos/bcontroller.yml

killed the proc and PDU cycled.

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15938631&tree=Mozilla-Inbound

Comment 9

[[e-mail] (:bhearsum)]

This tegra is successfully completing jobs.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

We're hitting bug 832052 exclusively on tegra-103. Please take it out of slavealloc.

Comment 11

[Ed Morley [:emorley]]

Tegra kindly disabled by Callek.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Hit this twice today now only on tegra-103.
https://tbpl.mozilla.org/php/getParsedLog.php?id=23777280&tree=Mozilla-Inbound

0 INFO SimpleTest START
1 INFO TEST-START | testImportFromAndroid
EventExpecter: no longer listening for Gecko:Ready
INFO | automation.py | Application ran for: 0:00:13.256867
INFO | zombiecheck | Reading PID log: /tmp/tmpQtkaq0pidlog
/data/anr/traces.txt not found
PROCESS-CRASH | java-exception | java.lang.NoSuchMethodError: ex035

Comment 13

[Armen [:armenzg]]

# On foopy116
touch /builds/tegra-103/disabled.flg

Comment 14

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 15

[Justin Wood (:Callek)]

Successfully recovered.

Comment 16

[Ed Morley [:emorley]]

Perma-retrying; disabled.

python /builds/sut_tools/installApp.py 10.26.85.79 build/fennec-32.0a1.en-US.android-arm.apk org.mozilla.fennec
 in dir /builds/tegra-103/test/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', '/builds/sut_tools/installApp.py', '10.26.85.79', u'build/fennec-32.0a1.en-US.android-arm.apk', 'org.mozilla.fennec']
 environment:
  HOME=/home/cltbld
  LOGNAME=cltbld
  OLDPWD=/home/cltbld
  PATH=/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/tegra-103/test
  PYTHONPATH=/builds/sut_tools
  SHELL=/bin/sh
  SHLVL=4
  SUT_IP=10.26.85.79
  SUT_NAME=tegra-103
  USER=cltbld
  _=/tools/buildbot/bin/python2.7
 using PTY: False
05/27/2014 03:28:18: INFO: copying build/fennec/application.ini to build/talos/remoteapp.ini
05/27/2014 03:28:18: DEBUG: calling [cp build/fennec/application.ini build/talos/remoteapp.ini]
05/27/2014 03:28:18: DEBUG: cp: cannot create regular file `build/talos/remoteapp.ini': No such file or directory
05/27/2014 03:28:18: INFO: connecting to: 10.26.85.79
reconnecting socket
05/27/2014 03:28:19: INFO: devroot /mnt/sdcard/tests
05/27/2014 03:28:19: INFO: 10.26.84.16, 50103
05/27/2014 03:28:19: INFO: Current device time is 2014/05/27 03:28:17
05/27/2014 03:28:19: INFO: Setting device time to 2014/05/27 03:28:19
05/27/2014 03:28:19: INFO: Current device time is 2014/05/27 03:28:19
results: {'process': [['10007', '1213', 'com.android.inputmethod.latin'], ['1001', '1221', 'com.android.phone'], ['1000', '1020', 'system'], ['10029', '1343', 'com.android.deskclock'], ['10031', '1507', 'com.mozilla.SUTAgentAndroid'], ['10018', '1227', 'com.android.launcher'], ['10013', '1482', 'com.cooliris.media'], ['10004', '1375', 'android.process.media'], ['10009', '1438', 'com.android.quicksearchbox'], ['10002', '1474', 'com.android.music'], ['10032', '1428', 'com.mozilla.watcher'], ['10006', '1409', 'com.android.mms'], ['10010', '1394', 'com.android.providers.calendar'], ['10014', '1369', 'com.android.email'], ['10017', '1358', 'com.android.bluetooth'], ['1000', '1239', 'com.android.settings'], ['10015', '1279', 'android.process.acore']]}
results: {'memory': ['PA:829992960, FREE: 754614272']}
results: {'uptime': ['0 days 0 hours 9 minutes 53 seconds 610 ms']}
results: {'screen': ['X:1024 Y:768']}
05/27/2014 03:28:19: INFO: Installing /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
in push file with: build/fennec-32.0a1.en-US.android-arm.apk, and: /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
sending: push /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
push returned: 71cd9035fd18a19bb3f545d719d67f65
Push File Validated!
05/27/2014 03:28:38: INFO: /builds/tegra-103/test/../error.flg
Remote Device Error: updateApp() call failed - exiting
program finished with exit code 1
elapsedTime=49.949731

Comment 17

[Van Le [:van]]

replaced SD card, flashed and reimaged tegra.

vle@vle-10516 ~ $ telnet tegra-103.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.79...
Connected to tegra-103.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q