Bug 722063 (tegra-103)

tegra-103 problem tracking

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
7 years ago
7 months ago

People

(Reporter: philor, Unassigned)

Tracking

Details

(Whiteboard: [badslave][buildduty], URL)

(Reporter)

Description

7 years ago
Looks like it broke on the morning of November 21st, since which time it has had 970 exception and retry runs in a row.
Ran stop_cp, clientproxy and buildslave are offline.
Priority: -- → P3
Assignee: nobody → bhearsum
(Reporter)

Comment 2

7 years ago
Looks like someone reenabled it (or it can zombie) on February 13th, on which lucky day it did four green runs and then four red/purple. Then it got reenabled this morning around 10, did 7 green runs, then timed out running tests in https://tbpl.mozilla.org/php/getParsedLog.php?id=9834484&tree=Try (a typical bad-tegra thing to do, I'm starting to realize), then timed out in cleanup device in https://tbpl.mozilla.org/php/getParsedLog.php?id=9835666&tree=Mozilla-Inbound and if it stays true to form, will continue to fail now until it's disabled and reimaged (or disabled and put in staging, or disabled and thrown out back, not sure what happens after the part I like, "disabled").
Summary: tegra-103 needs help → Please disable tegra-103 and enroll it in tegra recovery
'stop_cp tegra-103' has been run on foopy12.
Alias: tegra-103
Summary: Please disable tegra-103 and enroll it in tegra recovery → tegra-103 problem tracking
Assignee: bhearsum → nobody
Component: Release Engineering → Release Engineering: Machine Management
QA Contact: release → armenzg
Back in production.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
dead again... PDU cycles are not even bringing up the telnet port we need. Please reimage.
Status: RESOLVED → REOPENED
Depends on: 780798
Resolution: FIXED → ---
Apparantly it was just taking extra long to come up, working fine now.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago6 years ago
Resolution: --- → FIXED
No longer depends on: 780798
Last job 9 days, 19:47:52 ago

Could connect via telnet, no prompt

Also had a stale proc:
cltbld   32540   0.0  0.1  2456664  10468   ??  S    28Sep12   0:00.58 /opt/local/Library/Frameworks/Python.frame
work/Versions/2.6/Resources/Python.app/Contents/MacOS/Python /builds/tegra-103/talos-data/talos/bcontroller.py --
configFile /builds/tegra-103/talos-data/talos/bcontroller.yml

killed the proc and PDU cycled.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
This tegra is successfully completing jobs.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
We're hitting bug 832052 exclusively on tegra-103. Please take it out of slavealloc.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Updated

6 years ago
Depends on: 832052
Tegra kindly disabled by Callek.
Depends on: 877722
Hit this twice today now only on tegra-103.
https://tbpl.mozilla.org/php/getParsedLog.php?id=23777280&tree=Mozilla-Inbound

0 INFO SimpleTest START
1 INFO TEST-START | testImportFromAndroid
EventExpecter: no longer listening for Gecko:Ready
INFO | automation.py | Application ran for: 0:00:13.256867
INFO | zombiecheck | Reading PID log: /tmp/tmpQtkaq0pidlog
/data/anr/traces.txt not found
PROCESS-CRASH | java-exception | java.lang.NoSuchMethodError: ex035

Comment 13

6 years ago
# On foopy116
touch /builds/tegra-103/disabled.flg
Sending this slave to recovery
-->Automated message.
Depends on: 889567
Successfully recovered.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Perma-retrying; disabled.

python /builds/sut_tools/installApp.py 10.26.85.79 build/fennec-32.0a1.en-US.android-arm.apk org.mozilla.fennec
 in dir /builds/tegra-103/test/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', '/builds/sut_tools/installApp.py', '10.26.85.79', u'build/fennec-32.0a1.en-US.android-arm.apk', 'org.mozilla.fennec']
 environment:
  HOME=/home/cltbld
  LOGNAME=cltbld
  OLDPWD=/home/cltbld
  PATH=/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/tegra-103/test
  PYTHONPATH=/builds/sut_tools
  SHELL=/bin/sh
  SHLVL=4
  SUT_IP=10.26.85.79
  SUT_NAME=tegra-103
  USER=cltbld
  _=/tools/buildbot/bin/python2.7
 using PTY: False
05/27/2014 03:28:18: INFO: copying build/fennec/application.ini to build/talos/remoteapp.ini
05/27/2014 03:28:18: DEBUG: calling [cp build/fennec/application.ini build/talos/remoteapp.ini]
05/27/2014 03:28:18: DEBUG: cp: cannot create regular file `build/talos/remoteapp.ini': No such file or directory
05/27/2014 03:28:18: INFO: connecting to: 10.26.85.79
reconnecting socket
05/27/2014 03:28:19: INFO: devroot /mnt/sdcard/tests
05/27/2014 03:28:19: INFO: 10.26.84.16, 50103
05/27/2014 03:28:19: INFO: Current device time is 2014/05/27 03:28:17
05/27/2014 03:28:19: INFO: Setting device time to 2014/05/27 03:28:19
05/27/2014 03:28:19: INFO: Current device time is 2014/05/27 03:28:19
results: {'process': [['10007', '1213', 'com.android.inputmethod.latin'], ['1001', '1221', 'com.android.phone'], ['1000', '1020', 'system'], ['10029', '1343', 'com.android.deskclock'], ['10031', '1507', 'com.mozilla.SUTAgentAndroid'], ['10018', '1227', 'com.android.launcher'], ['10013', '1482', 'com.cooliris.media'], ['10004', '1375', 'android.process.media'], ['10009', '1438', 'com.android.quicksearchbox'], ['10002', '1474', 'com.android.music'], ['10032', '1428', 'com.mozilla.watcher'], ['10006', '1409', 'com.android.mms'], ['10010', '1394', 'com.android.providers.calendar'], ['10014', '1369', 'com.android.email'], ['10017', '1358', 'com.android.bluetooth'], ['1000', '1239', 'com.android.settings'], ['10015', '1279', 'android.process.acore']]}
results: {'memory': ['PA:829992960, FREE: 754614272']}
results: {'uptime': ['0 days 0 hours 9 minutes 53 seconds 610 ms']}
results: {'screen': ['X:1024 Y:768']}
05/27/2014 03:28:19: INFO: Installing /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
in push file with: build/fennec-32.0a1.en-US.android-arm.apk, and: /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
sending: push /mnt/sdcard/tests/fennec-32.0a1.en-US.android-arm.apk
push returned: 71cd9035fd18a19bb3f545d719d67f65
Push File Validated!
05/27/2014 03:28:38: INFO: /builds/tegra-103/test/../error.flg
Remote Device Error: updateApp() call failed - exiting
program finished with exit code 1
elapsedTime=49.949731
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Updated

5 years ago
QA Contact: armenzg → bugspam.Callek
Depends on: 1031502
No longer depends on: 832052

Comment 17

5 years ago
replaced SD card, flashed and reimaged tegra.

vle@vle-10516 ~ $ telnet tegra-103.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.79...
Connected to tegra-103.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q
(Reporter)

Updated

5 years ago
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → FIXED

Updated

7 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.