Closed Bug 807230 Opened 8 years ago Closed 6 years ago

Intermittent DMError: Automation Error: Timeout in command {ls,ps,mkdr,rmdir,activity} | DMError: Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs (or DMError: Automation Error: Timeout in command isdir /mnt/sdcard/tests/reftest)

Categories

(Release Engineering :: General, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: philor, Unassigned)

References

Details

https://tbpl.mozilla.org/php/getParsedLog.php?id=16608771&tree=Mozilla-Inbound
Android Tegra 250 mozilla-inbound opt test mochitest-1 on 2012-10-30 20:10:49 PDT for push a2afde356192
slave: tegra-228

43453 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | [conformance/renderbuffers/framebuffer-object-attachment.html] (WebGL mochitest) Starting test page
INFO | runtests.py | Received unexpected exception while running application
Traceback (most recent call last):
  File "/builds/tegra-228/test/build/tests/mochitest/runtests.py", line 731, in runTests
    timeout = timeout)
  File "/builds/tegra-228/test/build/tests/mochitest/automation.py", line 1050, in runApp
    status = self.waitForFinish(proc, utilityPath, timeout, maxTime, startTime, debuggerInfo, symbolsPath)
  File "/builds/tegra-228/test/build/tests/mochitest/remoteautomation.py", line 66, in waitForFinish
    status = proc.wait(timeout = maxTime)
  File "/builds/tegra-228/test/build/tests/mochitest/remoteautomation.py", line 203, in wait
    t = self.stdout
  File "/builds/tegra-228/test/build/tests/mochitest/remoteautomation.py", line 180, in stdout
    if self.dm.fileExists(self.proc):
  File "/builds/tegra-228/test/build/tests/mochitest/devicemanagerSUT.py", line 407, in fileExists
    return s[-1] in self.listFiles(containingpath)
  File "/builds/tegra-228/test/build/tests/mochitest/devicemanagerSUT.py", line 418, in listFiles
    data = self._runCmds([{ 'cmd': 'cd ' + rootdir }, { 'cmd': 'ls' }])
  File "/builds/tegra-228/test/build/tests/mochitest/devicemanagerSUT.py", line 140, in _runCmds
    self._sendCmds(cmdlist, outputfile, timeout)
  File "/builds/tegra-228/test/build/tests/mochitest/devicemanagerSUT.py", line 122, in _sendCmds
    raise err
DMError: Automation Error: Timeout in command ls
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

INFO | runtests.py | Running tests: end.
TEST-UNEXPECTED-FAIL | Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs | Exception caught while running tests.
WARNING: Error getting device information at end of test
program finished with exit code 1
Whiteboard: [orange]
Depends on: 816581
Summary: Intermittent DMError: Automation Error: Timeout in command ls, TEST-UNEXPECTED-FAIL | Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs | Exception caught while running tests. → Intermittent DMError: Automation Error: Timeout in command {ls,ps}, TEST-UNEXPECTED-FAIL | Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs | Exception caught while running tests.
this looks to be we are executing a command and the device becomes unreachable.  Mostly evident by the fact that we get secondary errors about no route to host, connection refused, etc...
Summary: Intermittent DMError: Automation Error: Timeout in command {ls,ps}, TEST-UNEXPECTED-FAIL | Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs | Exception caught while running tests. → Intermittent DMError: Automation Error: Timeout in command {ls,ps,isdir}, TEST-UNEXPECTED-FAIL | Automation Error: Timeout in command isdir /mnt/sdcard/tests/logs | Exception caught while running tests.
(In reply to Joel Maher (:jmaher) from comment #96)
> this looks to be we are executing a command and the device becomes
> unreachable.  Mostly evident by the fact that we get secondary errors about
> no route to host, connection refused, etc...

Do we have any next steps? :-)
I suspect this is related to the libc.so fix we did where we would hang after doing shell commands in the sutagent.

One thing to look into would be updating the sutagent and/or automation scripts to do as few shell commands as possible.  We should only be doing it a few times per test run, so if we are executing commands to get status, stats, or logs which do this we should find other ways to get that data or find ways to not require that data so frequently.