Closed Bug 848540 Opened 12 years ago Closed 9 years ago

tegra boards do not come back with an SUT agent that sticks

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: armenzg, Unassigned)

Details

Attachments

(1 file)

tegra-337.log
104.86 KB, text/plain
Details
Attached file tegra-337.log —
I have noticed this on tegras 360 and 337. The message on the error.flg is this: Armens-MacBook-Air ~ $ ssh foopy31 "tail /builds/tegra-337/error.flg" Automation Error: Unable to connect to device after 5 attempts How important is to write to /data/local/tmp ? I can't find anything interesting on the log. Armens-MacBook-Air ~ $ grep "SUT" tegra-337.log 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.REBOOT to package com.mozilla.SUTAgentAndroid (protectionLevel=3 flags=0xbe46) 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.DEVICE_POWER to package com.mozilla.SUTAgentAndroid (protectionLevel=2 flags=0xbe46) 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.INSTALL_PACKAGES to package com.mozilla.SUTAgentAndroid (protectionLevel=3 flags=0xbe46) 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.STATUS_BAR to package com.mozilla.SUTAgentAndroid (protectionLevel=3 flags=0xbe46) 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.SET_TIME to package com.mozilla.SUTAgentAndroid (protectionLevel=3 flags=0xbe46) 01-01 00:00:46.724 W/PackageManager( 1007): Not granting permission android.permission.WRITE_SECURE_SETTINGS to package com.mozilla.SUTAgentAndroid (protectionLevel=3 flags=0xbe46) 01-01 00:01:54.644 I/ActivityManager( 1007): Starting activity: Intent { act=android.intent.action.MAIN flg=0x10000000 pkg=com.mozilla.SUTAgentAndroid cmp=com.mozilla.SUTAgentAndroid/.SUTAgentAndroid } 01-01 00:01:54.724 I/ActivityManager( 1007): Start proc com.mozilla.SUTAgentAndroid for activity com.mozilla.SUTAgentAndroid/.SUTAgentAndroid: pid=1479 uid=10031 gids={3003, 1015, 3002, 3001} 01-01 00:01:54.834 I/SUTAgentAndroid( 1479): onCreate 01-01 00:01:54.854 I/SUTAgentAndroid( 1479): Caught exception creating file in /data/local/tmp: Permission denied 01-01 00:01:54.854 E/SUTAgentAndroid( 1479): ERROR: Cannot access world writeable test root 01-01 00:02:08.404 I/SUTAgentAndroid( 1479): CmdWorkerThread starts: 14 01-01 00:02:10.574 I/SUTAgentAndroid( 1479): 10.26.74.22 : ver 01-01 00:03:04.884 I/SUTAgentAndroid( 1479): CmdWorkerThread starts: 16 01-01 00:03:04.884 I/SUTAgentAndroid( 1479): 10.250.48.26 : testroot 01-01 00:03:04.894 I/SUTAgentAndroid( 1479): Caught exception creating file in /data/local/tmp: Permission denied 01-01 00:03:04.904 E/SUTAgentAndroid( 1479): ERROR: Cannot access world writeable test root 01-01 00:03:04.904 E/AndroidRuntime( 1479): at com.mozilla.SUTAgentAndroid.service.CmdWorkerThread.run(CmdWorkerThread.java:142) 01-01 00:03:04.914 W/ActivityManager( 1007): Force finishing activity com.mozilla.SUTAgentAndroid/.SUTAgentAndroid 01-01 00:03:05.034 I/SUTAgentAndroid( 1479): onDestroy - finishing 01-01 00:03:05.044 I/ActivityManager( 1007): Process com.mozilla.SUTAgentAndroid (pid 1479) has died. 01-01 00:03:05.044 I/WindowManager( 1007): WIN DEATH: Window{4856c878 com.mozilla.SUTAgentAndroid/com.mozilla.SUTAgentAndroid.SUTAgentAndroid paused=false} 01-01 00:03:15.034 W/ActivityManager( 1007): Activity destroy timeout for HistoryRecord{48574bc0 com.mozilla.SUTAgentAndroid/.SUTAgentAndroid} 01-01 00:06:54.644 I/ActivityManager( 1007): Starting activity: Intent { act=android.intent.action.MAIN flg=0x10000000 pkg=com.mozilla.SUTAgentAndroid cmp=com.mozilla.SUTAgentAndroid/.SUTAgentAndroid } 01-01 00:06:54.694 I/ActivityManager( 1007): Start proc com.mozilla.SUTAgentAndroid for activity com.mozilla.SUTAgentAndroid/.SUTAgentAndroid: pid=1546 uid=10031 gids={3003, 1015, 3002, 3001} 01-01 00:06:54.794 I/SUTAgentAndroid( 1546): onCreate 01-01 00:06:54.814 I/SUTAgentAndroid( 1546): Caught exception creating file in /data/local/tmp: Permission denied 01-01 00:06:54.814 E/SUTAgentAndroid( 1546): ERROR: Cannot access world writeable test root 01-01 00:08:53.704 I/SUTAgentAndroid( 1546): CmdWorkerThread starts: 15 01-01 00:09:11.574 I/SUTAgentAndroid( 1546): 10.26.74.22 : cd /data/data/com.mozilla.watcher/files 01-01 00:09:12.834 I/SUTAgentAndroid( 1546): 10.26.74.22 : ls 01-01 00:09:18.994 I/SUTAgentAndroid( 1546): 10.26.74.22 : cat watcher.ini 01-01 00:09:51.284 I/SUTAgentAndroid( 1546): CmdWorkerThread starts: 16 01-01 00:09:51.294 I/SUTAgentAndroid( 1546): CmdWorkerThread ends: 16 01-01 00:09:58.674 I/SUTAgentAndroid( 1546): 10.26.74.22 : adb ip 01-01 00:10:44.544 I/SUTAgentAndroid( 1546): 10.26.74.22 : exec su -c "logcat -d -v time *:W" 01-01 00:12:01.704 I/SUTAgentAndroid( 1546): 10.26.74.22 : quit 01-01 00:12:01.774 I/SUTAgentAndroid( 1546): CmdWorkerThread ends: 15 01-01 00:12:09.324 I/SUTAgentAndroid( 1546): CmdWorkerThread starts: 22 01-01 00:12:12.524 I/SUTAgentAndroid( 1546): 10.26.74.22 : adb ip 01-01 00:16:25.144 I/SUTAgentAndroid( 1546): 10.26.74.22 : exec su -c "logcat -d -v time *:W 01-01 00:16:29.934 I/SUTAgentAndroid( 1546): 10.26.74.22 : exec su -c "logcat -d -v time *:W"
When I manage to connect to the device I fail to write to it. $>exec sh -c 'echo foobar > /mnt/sdcard/foobar' cannot create /mnt/sdcard/foobar: permission denied return code [2] $>exec sh -c 'echo foobar > /data/local/tmp/foobar' cannot create /data/local/tmp/foobar: permission denied return code [2] I have tried my format_sd_card.py script (https://bugzilla.mozilla.org/attachment.cgi?id=717293) and also tried formatting remotely but we can't do anything. I'm going to keep on looking for more boards with the same error.flg and report back. $>exec mount rootfs / rootfs ro,relatime 0 0 tmpfs /dev tmpfs rw,relatime,mode=755 0 0 devpts /dev/pts devpts rw,relatime,mode=600 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,relatime 0 0 tmpfs /mnt/asec tmpfs rw,relatime,mode=755,gid=1000 0 0 /dev/block/mtdblock3 /system yaffs2 ro,relatime 0 0 /dev/block/mtdblock6 /data yaffs2 rw,nosuid,nodev,relatime 0 0 /dev/block/mtdblock4 /cache yaffs2 rw,nosuid,nodev,relatime 0 0 /dev/block/mtdblock0 /misc yaffs2 rw,nosuid,nodev,relatime 0 0 return code [0] $>exec newfs_msdos -F 32 /dev/block/vold/179:9 newfs_msdos: /dev/block/vold/179:9: No such file or directory return code [1] $>exec rebt $>^] telnet> quit Connection closed. (venv)Armens-MacBook-Air sut_tools hg:[default!] $ python format_sd_card.py 03/06/2013 22:12:11: INFO: The device needs to be rebooted through the PDU (not through SUT) 03/06/2013 22:12:11: INFO: Starting new HTTPS connection (1): inventory.mozilla.org 03/06/2013 22:12:12: DEBUG: "GET /en-US/tasty/v3/system/?limit=100&hostname__startswith=tegra-360 HTTP/1.1" 200 None Running command: '/usr/bin/snmpset -c private pdu3.dcops.build.mtv1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.10 i 3' SNMPv2-SMI::enterprises.1718.3.2.3.1.11.1.1.10 = INTEGER: 3 03/06/2013 22:12:12: INFO: Waiting for device to come back... 03/06/2013 22:12:42: INFO: Try 1 reconnecting socket ^@Could not connect; sleeping for 5 seconds. reconnecting socket Could not connect; sleeping for 10 seconds. reconnecting socket Could not connect; sleeping for 15 seconds. ^@reconnecting socket Could not connect; sleeping for 20 seconds. reconnecting socket Traceback (most recent call last): File "format_sd_card.py", line 82, in <module> main() File "format_sd_card.py", line 48, in main result = _waitForDevice(device) File "format_sd_card.py", line 19, in _waitForDevice dm = DeviceManagerSUT(device) File "/Users/armenzg/repos/tools/sut_tools/mozdevice/devicemanagerSUT.py", line 56, in __init__ raise BaseException("Failed to connect to SUT Agent and retrieve the device root.") BaseException: Failed to connect to SUT Agent and retrieve the device root.
high possibility we have bad sdcards, can we reimage these systems and get new sdcards?
(In reply to Armen Zambrano G. [:armenzg] from comment #1) > $>exec sh -c 'echo foobar > /data/local/tmp/foobar' > cannot create /data/local/tmp/foobar: permission denied > return code [2] I vaguely remember us point /data/local/tmp at the sdcard so that we are sure it gets cleaned properly and doesn't fill up local system. I can help look into this. > $>exec mount This commands output alone shows no sdcard is mounted as far as Android knows. So its pretty clear *something* went wrong while we were mounting it. Or *after* we mounted it. Possibilities include: * Loose terminals in the sdcard reader; * bad SDCard; * broken sdcard reader leads to board; * bad OS image; * bad board. I suspect a reimage of OS + a replaced SDCard will be best here. (I note that the newer style sdcards are more flakey than the original tegra sdcards)
It seems that I have all of these boards on this same state: 2013-03-07 13:50:35 tegra-072 p INACTIVE active OFFLINE :: SUTAgent not present;error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-07 13:50:12 tegra-293 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-07 13:50:13 tegra-311 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-06 19:25:13 tegra-337 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-07 13:50:07 tegra-355 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-06 19:24:13 tegra-360 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-07 13:50:55 tegra-363 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 2013-03-07 13:51:07 tegra-368 p online active OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] I will add disabled.flg to them and deal with them tomorrow.
I used this to get the info: * Put in a file this line foopy07 "tail -n 300 /builds/tegra-032/tegra-032_status.log" | grep "error.flg" | tail -n 1 * Then run this: cat tegras.sorted.txt | xargs -L1 ssh Armens-MacBook-Air ~ $ ssh foopy10 "touch /builds/tegra-072/disabled.flg" Armens-MacBook-Air ~ $ ssh foopy22 "touch /builds/tegra-293/disabled.flg" Armens-MacBook-Air ~ $ ssh foopy28 "touch /builds/tegra-311/disabled.flg" Armens-MacBook-Air ~ $ ssh foopy32 "touch /builds/tegra-355/disabled.flg" Armens-MacBook-Air ~ $ ssh foopy32 "touch /builds/tegra-363/disabled.flg" Armens-MacBook-Air ~ $ ssh foopy32 "touch /builds/tegra-368/disabled.flg"
Depends on: 849381
No longer depends on: 849381
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: