890422 - XPCShell timeouts on Android retry infinitely

Status: RESOLVED DUPLICATE of bug 875814

(Testing :: XPCShell Harness, defect)

Description

[Phil Ringnalda (:philor)]

See, for example, https://tbpl.mozilla.org/?showall=1&tree=Gum&rev=bb8526f22ba4 or whatever the current tip of Gum is, if they haven't fixed their timeout, or https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=170e1863b9b6, or all the try pushes over the last few weeks where I've wrongly blamed the person who just pushed an innocent test failure.

https://tbpl.mozilla.org/php/getParsedLog.php?id=24957455&tree=Gum
Android Armv6 Tegra 250 gum opt test xpcshell on 2013-07-04 17:56:34 PDT for push 2dae8ffcab5e
slave: tegra-269

TEST-INFO | /builds/tegra-269/test/build/tests/xpcshell/tests/image/test/unit/test_private_channel.js | running test ...
Exception in thread Thread-54:
Traceback (most recent call last):
  File "/tools/python27/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/tools/python27/lib/python2.7/threading.py", line 755, in run
    self.function(*self.args, **self.kwargs)
  File "/builds/tegra-269/test/build/tests/xpcshell/runxpcshelltests.py", line 889, in <lambda>
    testTimer = Timer(HARNESS_TIMEOUT, lambda: self.testTimeout(name, proc.pid))
AttributeError: 'NoneType' object has no attribute 'pid'

Traceback (most recent call last):
  File "xpcshell/remotexpcshelltests.py", line 503, in <module>
    main()
  File "xpcshell/remotexpcshelltests.py", line 498, in main
    **options.__dict__):
  File "/builds/tegra-269/test/build/tests/xpcshell/runxpcshelltests.py", line 794, in runTests
    pStderr=pStderr, keep_going=keepGoing)
  File "/builds/tegra-269/test/build/tests/xpcshell/runxpcshelltests.py", line 901, in run_test
    stdout=pStdout, stderr=pStderr, env=self.env, cwd=test_dir)
  File "xpcshell/remotexpcshelltests.py", line 309, in launchProcess
    self.shellReturnCode = self.device.shell(cmd, f)
  File "/builds/tegra-269/test/build/tests/xpcshell/devicemanagerSUT.py", line 339, in shell
    raise DMError("Automation Error: Error finding end of line/return value when running '%s'" % cmdline)
devicemanager.DMError: Automation Error: Error finding end of line/return value when running '/data/local/xpcb/xpcw /mnt/sdcard/tests/xpcshell/image/test/unit -r /mnt/sdcard/tests/xpcshell/c/httpd.manifest --greomni /data/local/xpcb/fennec-25.0a1.en-US.android-arm-armv6.apk -m -n -s -e 'const _HTTPD_JS_PATH = "/mnt/sdcard/tests/xpcshell/c/httpd.js";' -e 'const _HEAD_JS_PATH = "/mnt/sdcard/tests/xpcshell/head.js";' -e 'const _TESTING_MODULES_DIR = "/mnt/sdcard/tests/xpcshell/m";' -f /mnt/sdcard/tests/xpcshell/head.js -e 'const _SERVER_ADDR = "localhost"' -e 'const _HEAD_FILES = [];' -e 'const _TAIL_FILES = [];' -e 'const _TEST_FILE = ["test_private_channel.js"];' -e '_execute_test(); quit(0);''

Comment 1

[Geoff Brown [:gbrown]]

I think this will turn out to be a dup of bug 875814, which addresses the "'NoneType' object has no attribute 'pid'" part of the error. I suspect that condition also gives rise to the Automation Error, which is treated as a retry condition rather than a test failure. IMO, retry is appropriate because the cause is a fault in the test scripts (referencing the non-existent pid field) rather than a test failure. 

The result is painful for the sheriffs and wasteful of tbpl resources: a test timeout will cause a retry which will have the same result, another retry, ... until someone cancels the job.

I will try to look at bug 875814 early next week.