1233454 - Intermittent "IndexError: pop from empty list" in devicemanagerADB.getProcessList

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Geoff Brown [:gbrown]]

This is probably hard to reproduce, but the vulnerability is apparent.

https://treeherder.mozilla.org/logviewer.html#?job_id=685196&repo=mozilla-beta

 16:25:52     INFO -  TEST-START | testing/xpcshell/example/unit/test_check_nsIException_failing.js
 16:30:52  WARNING -  TEST-UNEXPECTED-TIMEOUT | testing/xpcshell/example/unit/test_check_nsIException_failing.js | Test timed out
 16:30:52     INFO -  TEST-INFO expected FAIL | took 300000ms
 16:31:02     INFO -  mozdevice Timeout exceeded for shell call 'adb shell /data/local/xpcb/xpcw /sdcard/tests/xpcshell/testing/xpcshell/example/unit -r /sdcard/tests/xpcshell/c/httpd.manifest --greomni /data/local/xpcb/fennec-44.0.en-US.android-arm.apk -m -s -e 'const _HEAD_JS_PATH = "/sdcard/tests/xpcshell/head.js";' -e 'const _MOZINFO_JS_PATH = "/sdcard/tests/xpcshell/p/mozinfo.json";' -e 'const _TESTING_MODULES_DIR = "/sdcard/tests/xpcshell/m";' -f /sdcard/tests/xpcshell/head.js -e 'const _SERVER_ADDR = "localhost"' -e 'const _HEAD_FILES = [];' -e 'const _TAIL_FILES = [];' -e 'const _JSDEBUGGER_PORT = 0;' -e 'const _TEST_FILE = ["test_check_nsIException_failing.js"];' -e 'const _TEST_NAME = "testing/xpcshell/example/unit/test_check_nsIException_failing.js"' -e '_execute_test(); quit(0);'; echo $?'
 16:31:22     INFO -  mozdevice Timeout exceeded for _runCmd call 'adb shell ps'
 16:31:22     INFO -  Exception in thread Thread-1176:
 16:31:22     INFO -  Traceback (most recent call last):
 16:31:22     INFO -    File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
 16:31:22     INFO -      self.run()
 16:31:22     INFO -    File "/usr/lib/python2.7/threading.py", line 755, in run
 16:31:22     INFO -      self.function(*self.args, **self.kwargs)
 16:31:22     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 661, in <lambda>
 16:31:22     INFO -      testTimer = Timer(testTimeoutInterval, lambda: self.testTimeout(proc))
 16:31:22     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 309, in testTimeout
 16:31:22     INFO -      self.killTimeout(proc)
 16:31:22     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 133, in killTimeout
 16:31:22     INFO -      self.kill(proc)
 16:31:22     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 189, in kill
 16:31:22     INFO -      return self.device.killProcess("xpcshell", True)
 16:31:22     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozdevice/devicemanagerADB.py", line 389, in killProcess
 16:31:22     INFO -      procs = self.getProcessList()
 16:31:22     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozdevice/devicemanagerADB.py", line 313, in getProcessList
 16:31:22     INFO -      p.output.pop(0)
 16:31:22     INFO -  IndexError: pop from empty list
 16:31:32     INFO -  mozdevice Timeout exceeded for _runCmd call 'adb shell ps'
 16:31:32     INFO -  Following exceptions were raised:
 16:31:32     INFO -  Traceback (most recent call last):
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 165, in run
 16:31:32     INFO -      self.run_test()
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 673, in run_test
 16:31:32     INFO -      stdout=self.pStdout, stderr=self.pStderr, env=self.env, cwd=test_dir, timeout=testTimeoutInterval)
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 155, in launchProcess
 16:31:32     INFO -      self.device.killProcess("xpcshell")
 16:31:32     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozdevice/devicemanagerADB.py", line 389, in killProcess
 16:31:32     INFO -      procs = self.getProcessList()
 16:31:32     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/mozdevice/devicemanagerADB.py", line 313, in getProcessList
 16:31:32     INFO -      p.output.pop(0)
 16:31:32     INFO -  IndexError: pop from empty list
 16:31:32     INFO -  Traceback (most recent call last):
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 623, in <module>
 16:31:32     INFO -      main()
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 618, in main
 16:31:32     INFO -      **vars(options)):
 16:31:32     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 1408, in runTests
 16:31:32     INFO -      raise exceptions[0]
 16:31:32     INFO -  IndexError: pop from empty list
 16:31:33    ERROR - Return code: 1
 16:31:33    ERROR - No tests run or test summary not found

Comment 1

[Geoff Brown [:gbrown]]

Attachment: guard against ps failure in getProcessList

Does no harm: https://treeherder.mozilla.org/#/jobs?repo=try&revision=69d2c1f3cb05&selectedJob=14703481

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8699567 [details] [diff] [review]
guard against ps failure in getProcessList

Review of attachment 8699567 [details] [diff] [review]:
-----------------------------------------------------------------

you keep finding these simple bugs!

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/cf947522f851

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/cf947522f851