Closed Bug 822555 Opened 13 years ago Closed 13 years ago

Consistent failure on b2g18 in "TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/test/unit/test_0114_general.js | test failed (with xpcshell return code: -1)"

Categories

(Testing :: XPCShell Harness, defect)

Product:
Testing
Component:
XPCShell Harness
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: cjones, Unassigned)

References

Details

https://tbpl.mozilla.org/php/getParsedLog.php?id=18037035&tree=Mozilla-B2g18#error0 b2g_ics_armv7a_gecko_emulator mozilla-b2g18 opt test xpcshell on 2012-12-17 18:28:03 PST for push 41e791c6b3ae slave: talos-r3-fed-007 18:49:19 WARNING - TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/tests/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0114_general.js | test failed (with xpcshell return code: -1), see following log: 18:50:06 ERROR - F/libc ( 43): Fatal signal 11 (SIGSEGV) at 0x00000047 (code=1) 18:50:06 ERROR - This usually indicates the B2G process has crashed (These two lines are misleading, not related to timeout.) The real failure is 18:49:19 WARNING - This is a harness error. 18:49:19 INFO - >>>>>>> 18:49:19 INFO - Traceback (most recent call last): 18:49:19 INFO - File "runtestsb2g.py", line 71, in launchProcess 18:49:19 INFO - outputFile = XPCShellRemote.launchProcess(self, cmd, stdout, stderr, env, cwd) 18:49:19 INFO - File "/home/cltbld/talos-slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 230, in launchProcess 18:49:19 INFO - self.shellReturnCode = self.device.shell(cmd, f, cwd=self.remoteHere, env=env) 18:49:19 INFO - File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozdevice/devicemanagerADB.py", line 138, in shell 18:49:19 INFO - raise DMError("Timeout exceeded for shell call") 18:49:19 INFO - DMError: Timeout exceeded for shell call 18:49:19 INFO - <<<<<<< Anyone know what might be up here? We can try disabling this test for now.
It seems like bug 821343 might have started this just by enabling the tests :/. https://hg.mozilla.org/releases/mozilla-b2g18/rev/a7e5e06a64ed
This test is passing on m-c consistently as well as during my try pushes testing the xpcshell tests on b2g.
We have two green runs on aurora too. Are we missing any patches on b2g18?
I'll clone mozilla-b2g18 and diff toolkit/mozapps/update/
Summary: Consistent failure in "TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/test/unit/test_0114_general.js | test failed (with xpcshell return code: -1)" → Consistent failure on b2g18 in "TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/test/unit/test_0114_general.js | test failed (with xpcshell return code: -1)"
The only toolkit/mozapps/update bugs landed on m-c that might affect this though I think it is unlikely are Bug 794234 Bug 765598 Bug 805466
There are significant differences in remotexpcshelltests.py --- testing/xpcshell/remotexpcshelltests.py 2012-12-17 23:05:22 -0800 +++ ../../_1_mozilla-central/mozilla/testing/xpcshell/remotexpcshelltests.py 2012-12-07 16:42:39 -0800 @@ -2,16 +2,17 @@ # # This Source Code Form is subject to the terms of the Mozilla Public # License, v. 2.0. If a copy of the MPL was not distributed with this # file, You can obtain one at http://mozilla.org/MPL/2.0/. import sys, os import subprocess import runxpcshelltests as xpcshell +import tempfile from automationutils import replaceBackSlashes from mozdevice import devicemanagerADB, devicemanagerSUT, DMError here = os.path.dirname(os.path.abspath(__file__)) # A specialization of XPCShellTests that runs tests on an Android device # via devicemanager. class XPCShellRemote(xpcshell.XPCShellTests, object): @@ -28,30 +29,33 @@ # to minimize the length of the command line used to execute # xpcshell on the remote device. adb has a limit to the number # of characters used in a shell command, and the xpcshell command # line can be quite complex. self.remoteBinDir = self.remoteJoin(self.remoteTestRoot, "b") self.remoteTmpDir = self.remoteJoin(self.remoteTestRoot, "tmp") self.remoteScriptsDir = self.remoteTestRoot self.remoteComponentsDir = self.remoteJoin(self.remoteTestRoot, "c") + self.remoteModulesDir = self.remoteJoin(self.remoteTestRoot, "m") self.profileDir = self.remoteJoin(self.remoteTestRoot, "p") self.remoteDebugger = options.debugger self.remoteDebuggerArgs = options.debuggerArgs + self.testingModulesDir = options.testingModulesDir if self.options.objdir: self.xpcDir = os.path.join(self.options.objdir, "_tests/xpcshell") elif os.path.isdir(os.path.join(here, 'tests')): self.xpcDir = os.path.join(here, 'tests') else: print >> sys.stderr, "Couldn't find local xpcshell test directory" sys.exit(1) if options.setup: self.setupUtilities() + self.setupModules() self.setupTestDir() if options.localAPK: self.remoteAPK = self.remoteJoin(self.remoteBinDir, os.path.basename(options.localAPK)) self.setAppRoot() def setAppRoot(self): # Determine the application root directory associated with the package # name used by the Fennec APK. @@ -106,39 +110,42 @@ remoteFile = self.remoteJoin(self.remoteComponentsDir, "httpd.manifest") self.device.pushFile(local, remoteFile) local = os.path.join(self.localBin, "components/test_necko.xpt") remoteFile = self.remoteJoin(self.remoteComponentsDir, "test_necko.xpt") self.device.pushFile(local, remoteFile) if self.options.localAPK: - remoteFile = self.remoteJoin(self.remoteBinDir, os.path.basename(self.options.localAPK)) - self.device.pushFile(self.options.localAPK, remoteFile) + remoteFile = self.remoteJoin(self.remoteBinDir, os.path.basename(self.options.localAPK)) + self.device.pushFile(self.options.localAPK, remoteFile) self.pushLibs() - self.device.chmodDir(self.remoteBinDir) - def pushLibs(self): for file in os.listdir(self.localLib): if (file.endswith(".so")): print >> sys.stderr, "Pushing %s.." % file if 'libxul' in file: print >> sys.stderr, "This is a big file, it could take a while." - self.device.pushFile(os.path.join(self.localLib, file), self.remoteBinDir) + remoteFile = self.remoteJoin(self.remoteBinDir, file) + self.device.pushFile(os.path.join(self.localLib, file), remoteFile) # Additional libraries may be found in a sub-directory such as "lib/armeabi-v7a" localArmLib = os.path.join(self.localLib, "lib") if os.path.exists(localArmLib): for root, dirs, files in os.walk(localArmLib): for file in files: if (file.endswith(".so")): - self.device.pushFile(os.path.join(root, file), self.remoteBinDir) + remoteFile = self.remoteJoin(self.remoteBinDir, file) + self.device.pushFile(os.path.join(root, file), remoteFile) + def setupModules(self): + if self.testingModulesDir: + self.device.pushDir(self.testingModulesDir, self.remoteModulesDir) def setupTestDir(self): push_attempts = 10 for retry in range(1, push_attempts+1): print 'pushing', self.xpcDir, '(attempt %s of %s)' % (retry, push_attempts) try: self.device.pushDir(self.xpcDir, self.remoteScriptsDir) break @@ -152,27 +159,29 @@ for test in self.alltests: uniqueTestPaths.add(test['here']) for testdir in uniqueTestPaths: abbrevTestDir = os.path.relpath(testdir, self.xpcDir) remoteScriptDir = self.remoteJoin(self.remoteScriptsDir, abbrevTestDir) self.pathMapping.append(PathMapping(testdir, remoteScriptDir)) def buildXpcsCmd(self, testdir): - self.xpcsCmd = [ - self.remoteJoin(self.remoteBinDir, "xpcshell"), - '-r', self.remoteJoin(self.remoteComponentsDir, 'httpd.manifest')] - # If using an APK, --greomni must be specified before any -e arguments. + # change base class' paths to remote paths and use base class to build command + self.xpcshell = self.remoteJoin(self.remoteBinDir, "xpcw") + self.headJSPath = self.remoteJoin(self.remoteScriptsDir, 'head.js') + self.httpdJSPath = self.remoteJoin(self.remoteComponentsDir, 'httpd.js') + self.httpdManifest = self.remoteJoin(self.remoteComponentsDir, 'httpd.manifest') + self.testingModulesDir = self.remoteModulesDir + self.testharnessdir = self.remoteScriptsDir + xpcshell.XPCShellTests.buildXpcsCmd(self, testdir) + # remove "-g <dir> -a <dir>" and add "--greomni <apk>" + del(self.xpcsCmd[1:5]) if self.options.localAPK: - self.xpcsCmd.extend(['--greomni', self.remoteAPK]) - self.xpcsCmd.extend([ - '-s', - '-e', 'const _HTTPD_JS_PATH = "%s";' % self.remoteJoin(self.remoteComponentsDir, 'httpd.js'), - '-e', 'const _HEAD_JS_PATH = "%s";' % self.remoteJoin(self.remoteScriptsDir, 'head.js'), - '-f', self.remoteScriptsDir+'/head.js']) + self.xpcsCmd.insert(3, '--greomni') + self.xpcsCmd.insert(4, self.remoteAPK) if self.remoteDebugger: # for example, "/data/local/gdbserver" "localhost:12345" self.xpcsCmd = [ self.remoteDebugger, self.remoteDebuggerArgs, self.xpcsCmd] @@ -207,32 +216,68 @@ def setupProfileDir(self): self.device.removeDir(self.profileDir) self.device.mkDir(self.profileDir) if self.interactive or self.singleFile: self.log.info("TEST-INFO | profile dir is %s" % self.profileDir) return self.profileDir + def logCommand(self, name, completeCmd, testdir): + self.log.info("TEST-INFO | %s | full command: %r" % (name, completeCmd)) + self.log.info("TEST-INFO | %s | current directory: %r" % (name, self.remoteHere)) + self.log.info("TEST-INFO | %s | environment: %s" % (name, self.env)) + + def setupLeakLogging(self): + self.env["XPCOM_MEM_LEAK_LOG"] = self.remoteJoin(self.profileDir, "leaks.log") + return self.env["XPCOM_MEM_LEAK_LOG"] + def setLD_LIBRARY_PATH(self, env): env["LD_LIBRARY_PATH"]=self.remoteBinDir - def launchProcess(self, cmd, stdout, stderr, env, cwd): - cmd[0] = self.remoteJoin(self.remoteBinDir, "xpcshell") - env = {} - self.setLD_LIBRARY_PATH(env) - env["MOZ_LINKER_CACHE"]=self.remoteBinDir + def pushWrapper(self): + # Rather than executing xpcshell directly, this wrapper script is + # used. By setting environment variables and the cwd in the script, + # the length of the per-test command line is shortened. This is + # often important when using ADB, as there is a limit to the length + # of the ADB command line. + localWrapper = tempfile.mktemp() + f = open(localWrapper, "w") + f.write("#!/system/bin/sh\n") + for envkey, envval in self.env.iteritems(): + f.write("export %s=%s\n" % (envkey, envval)) + f.write("cd $1\n") + f.write("echo xpcw: cd $1\n") + f.write("shift\n") + f.write("echo xpcw: xpcshell \"$@\"\n") + f.write("%s/xpcshell \"$@\"\n" % self.remoteBinDir) + f.close() + remoteWrapper = self.remoteJoin(self.remoteBinDir, "xpcw") + self.device.pushFile(localWrapper, remoteWrapper) + os.remove(localWrapper) + self.device.chmodDir(self.remoteBinDir) + + def buildEnvironment(self): + self.env = {} + self.buildCoreEnvironment() + self.setLD_LIBRARY_PATH(self.env) + self.env["MOZ_LINKER_CACHE"]=self.remoteBinDir if self.options.localAPK and self.appRoot: - env["GRE_HOME"]=self.appRoot - env["XPCSHELL_TEST_PROFILE_DIR"]=self.profileDir - env["TMPDIR"]=self.remoteTmpDir - env["HOME"]=self.profileDir + self.env["GRE_HOME"]=self.appRoot + self.env["XPCSHELL_TEST_PROFILE_DIR"]=self.profileDir + self.env["TMPDIR"]=self.remoteTmpDir + self.env["HOME"]=self.profileDir + if self.options.setup: + self.pushWrapper() + + def launchProcess(self, cmd, stdout, stderr, env, cwd): + cmd.insert(1, self.remoteHere) outputFile = "xpcshelloutput" f = open(outputFile, "w+") - self.shellReturnCode = self.device.shell(cmd, f, cwd=self.remoteHere, env=env) + self.shellReturnCode = self.device.shell(cmd, f) f.close() # The device manager may have timed out waiting for xpcshell. # Guard against an accumulation of hung processes by killing # them here. Note also that IPC tests may spawn new instances # of xpcshell. self.device.killProcess(cmd[0]) self.device.killProcess("xpcshell") return outputFile @@ -365,16 +410,27 @@ def __init__(self, localDir, remoteDir): self.local = localDir self.remote = remoteDir def main(): parser = RemoteXPCShellOptions() options, args = parser.parse_args() + if not options.localAPK: + for file in os.listdir(os.path.join(options.objdir, "dist")): + if (file.endswith(".apk") and file.startswith("fennec")): + options.localAPK = os.path.join(options.objdir, "dist") + options.localAPK = os.path.join(options.localAPK, file) + print >>sys.stderr, "using APK: " + options.localAPK + break + else: + print >>sys.stderr, "Error: please specify an APK" + sys.exit(1) + options = parser.verifyRemoteOptions(options) if len(args) < 1 and options.manifest is None: print >>sys.stderr, """Usage: %s <test dirs> or: %s --manifest=test.manifest """ % (sys.argv[0], sys.argv[0]) sys.exit(1) if (options.dm_trans == "adb"): @@ -391,28 +447,16 @@ if options.interactive and not options.testPath: print >>sys.stderr, "Error: You must specify a test filename in interactive mode!" sys.exit(1) if not options.objdir: print >>sys.stderr, "Error: You must specify an objdir" sys.exit(1) - if not options.localAPK: - for file in os.listdir(os.path.join(options.objdir, "dist")): - if (file.endswith(".apk") and file.startswith("fennec")): - options.localAPK = os.path.join(options.objdir, "dist") - options.localAPK = os.path.join(options.localAPK, file) - print >>sys.stderr, "using APK: " + options.localAPK - break - - if not options.localAPK: - print >>sys.stderr, "Error: please specify an APK" - sys.exit(1) - xpcsh = XPCShellRemote(dm, options, args) if not xpcsh.runTests(xpcshell='xpcshell', testdirs=args[0:], **options.__dict__): sys.exit(1)
I suspect this is due to bug 752126
Depends on: 752126
Good catch, that might be it. I'll see if I can reproduce this locally and try applying that patch. In the meantime I disabled the test (I'll re-enable with a working patch): https://hg.mozilla.org/releases/mozilla-b2g18/rev/57b016ab196e
So the changes in remotexpcshelltests.py that are missing in b2g18 are bug 808728, bug 808725, bug 752126, bug 808729, bug 808764 and bug 812191 (34a1715f5dd3 to cd88e0b232ba). I applied and unbitrotted all of these patches and got the same result, test_0114_general.js still times out.
I don't know what else could be different and with test_0114_general.js being the same on both m-c and b2g18 I highly doubt it is the test itself.
Maybe something in the updater is missing from b2g18. I'll see if I can figure out why it's timing out.
This might be the same problem that I'm seeing in bug 821782
See Also: → 821782
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.