Closed Bug 799863 Opened 12 years ago Closed 12 years ago

xpcshell-tests-remote not working properly

Categories

(Firefox for Android Graveyard :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gcp, Assigned: gbrown)

References

Details

Attachments

(4 files)

Following the instructions on the wiki: morbo@denebix:~/hg/mozilla-central/objdir-android$ make -C toolkit/components/telemetry/tests/ xpcshell-tests-remote make: Entering directory `/home/morbo/hg/mozilla-central/objdir-android/toolkit/components/telemetry/tests' /home/morbo/hg/mozilla-central/objdir-android/_virtualenv/bin/python -u /home/morbo/hg/mozilla-central/config/pythonpath.py \ -I/home/morbo/hg/mozilla-central/build \ -I/home/morbo/hg/mozilla-central/build/mobile \ /home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py \ --symbols-path=../../../../dist/crashreporter-symbols \ --build-info-json=../../../../mozinfo.json \ \ --dm_trans=adb \ --deviceIP= \ --objdir=../../../.. \ ../../../../_tests/xpcshell/toolkit/components/telemetry/tests/unit Traceback (most recent call last): File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 56, in <module> main(sys.argv[1:]) File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 48, in main execfile(script, frozenglobals) File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 11, in <module> import devicemanager,devicemanagerADB, devicemanagerSUT ImportError: No module named devicemanager make: *** [xpcshell-tests-remote] Error 1 The relevant python files are present for reftests and mochitests, but don't seem to be in the path for xpcshell-remote. After copying them manually to build/mobile: Traceback (most recent call last): File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 56, in <module> main(sys.argv[1:]) File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 48, in main execfile(script, frozenglobals) File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 368, in <module> main() File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 332, in main dm = devicemanagerADB.DeviceManagerADB() File "/home/morbo/hg/mozilla-central/build/mobile/devicemanagerADB.py", line 67, in __init__ self._verifyRunAs() File "/home/morbo/hg/mozilla-central/build/mobile/devicemanagerADB.py", line 879, in _verifyRunAs self._checkCmd(["push", tmpfile.name, tmpDir + "/tmpfile"]) TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' make: *** [xpcshell-tests-remote] Error 1
After some manual fiddling, I got a bit further. I think the instructions that mention making /data/local/tests also should mention that it probably needs to be chmod 777 or owned by the proper user.
It eventually fails here: chmod /data/local/tests/xpcshell/b/defaults/pref chmod /data/local/tests/xpcshell/b/defaults chmod /data/local/tests/xpcshell/b/fennec-18.0a1.en-US.android-arm.apk chmod /data/local/tests/xpcshell/b/libfreebl3.so chmod /data/local/tests/xpcshell/b/libmozalloc.so chmod /data/local/tests/xpcshell/b/libmozglue.so chmod /data/local/tests/xpcshell/b/libmozsqlite3.so chmod /data/local/tests/xpcshell/b/libnspr4.so chmod /data/local/tests/xpcshell/b/libnss3.so chmod /data/local/tests/xpcshell/b/libnssckbi.so chmod /data/local/tests/xpcshell/b/libnssutil3.so chmod /data/local/tests/xpcshell/b/libomxplugin.so chmod /data/local/tests/xpcshell/b/libplc4.so chmod /data/local/tests/xpcshell/b/libplds4.so chmod /data/local/tests/xpcshell/b/libplugin-container.so chmod /data/local/tests/xpcshell/b/libsmime3.so chmod /data/local/tests/xpcshell/b/libsoftokn3.so chmod /data/local/tests/xpcshell/b/libssl3.so chmod /data/local/tests/xpcshell/b/libxpcom.so chmod /data/local/tests/xpcshell/b/libxul.so chmod /data/local/tests/xpcshell/b/xpcshell chmod /data/local/tests/xpcshell/b Traceback (most recent call last): File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 56, in <module> main(sys.argv[1:]) File "/home/morbo/hg/mozilla-central/config/pythonpath.py", line 48, in main execfile(script, frozenglobals) File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 368, in <module> main() File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 363, in main **options.__dict__): File "/home/morbo/hg/mozilla-central/testing/xpcshell/runxpcshelltests.py", line 768, in runTests testHeadFiles, testTailFiles = self.getHeadAndTailFiles(test) File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 187, in getHeadAndTailFiles return (list(sanitize_list(test['head'], 'head')), File "/home/morbo/hg/mozilla-central/testing/xpcshell/remotexpcshelltests.py", line 181, in sanitize_list path)) Exception: head file does not exist: /data/local/tests/xpcshell/chrome/test/unit/head_crtestutils.js grep: xpcshell-tests-remote.log: No such file or directory /bin/sh: line 1: @errors=: command not found shell@android:/data/local/tests # busybox find . ./tmp ./xpcshell ./xpcshell/b ./xpcshell/b/defaults ./xpcshell/b/defaults/pref ./xpcshell/b/xpcshell ./xpcshell/b/fennec-18.0a1.en-US.android-arm.apk ./xpcshell/b/libssl3.so ./xpcshell/b/libplc4.so ./xpcshell/b/libnspr4.so ./xpcshell/b/libmozsqlite3.so ./xpcshell/b/libplds4.so ./xpcshell/b/libnssckbi.so ./xpcshell/b/libnssutil3.so ./xpcshell/b/libsoftokn3.so ./xpcshell/b/libnss3.so ./xpcshell/b/libfreebl3.so ./xpcshell/b/libmozalloc.so ./xpcshell/b/libxul.so ./xpcshell/b/libxpcom.so ./xpcshell/b/libsmime3.so ./xpcshell/b/libplugin-container.so ./xpcshell/b/libmozglue.so ./xpcshell/b/libomxplugin.so ./xpcshell/c ./xpcshell/c/httpd.js ./xpcshell/c/httpd.manifest ./xpcshell/c/test_necko.xpt ./xpcshell/head.js ./xpcshell/tmp
The path issue is probably just due to us not including mozdevice at its new location. I didn't notice that it needed to be moved because we don't run the XPCShell tests as part of tbpl, I guess. Something like the below should fix that issue. The errors copying the head file look more serious to me. I'll try to poke around and see what's going on. I hope that these tests weren't written with the assumption that adb was restarted in root mode or something... diff --git a/testing/testsuite-targets.mk b/testing/testsuite-targets.mk index 3804de8..f70673d 100644 --- a/testing/testsuite-targets.mk +++ b/testing/testsuite-targets.mk @@ -265,6 +265,7 @@ REMOTE_XPCSHELL = \ $(PYTHON) -u $(topsrcdir)/config/pythonpath.py \ -I$(topsrcdir)/build \ -I$(topsrcdir)/build/mobile \ + -I$(topsrcdir)/testing/mozbase/mozdevice/mozdevice \ $(topsrcdir)/testing/xpcshell/remotexpcshelltests.py \ --manifest=$(DEPTH)/_tests/xpcshell/xpcshell.ini \ --build-info-json=$(DEPTH)/mozinfo.json \
Ok, so on further investigation there are a whole bunch of problems with the XPC Shell tests on devicemanagerADB besides the path issue mentioned above: (1) Some of the adb methods for pushing files/directories were broken, probably by me unfortunately with bug (it's very difficult to test adb stuff as we have no kind of automated testing for it: I tried to test my patch with mochitest/reftest but forgot about xpcshell) (2) If /data/local/tests does not exist, we try to execute and run files from the sdcard. This will not work (you're not allowed to mark files as executable on the sdcard). Moreover, we have the problem that it is not possible to create /data/local/tests on an unrooted device (or at least it doesn't work on my Galaxy Nexus running 4.1) (3) On a device where root isn't the default user, we try to use run-as to make all the supporting files are owned by the fennec package. However, we execute the xpc shell tests as a different unprivileged user. This will likewise not work. If I hack past all that, the xpcshell tests do seem to run, though (perhaps unsurprisingly) one out of 4 of them fails. I think to get this working we need to (1) figure out what to do about the run-as problem and (2) determine what to do about unrooted devices. How high a priority is this? Were you trying to do something in particular?
I was trying to debug Telemetry. I worked around this by hacking around elsewhere and blocking the Telemetry from sending in a real build. So this isn't urgent at all. Being able to do xpcshell tests would be nice though, not all things can be easily debugged on try. I think the tests not working on unrooted devices is quite acceptable for development. Meanwhile, I'll point the wiki to this bug. https://wiki.mozilla.org/Mobile/Fennec/Android#xpcshell
Filed bug 800655 with patches for some of the adb issues mentioned above. Getting those in won't fix all the issues we're seeing here, but it's a start.
Depends on: 800655
Assignee: nobody → gbrown
This is blocking bug 806369 as it needs to be tested by a XPCshell test if we're to have any confidence in our ability to blacklist H.264 decoding with the AMO blacklist. I tried switching to the SUT devicemanager but that didn't make any difference. I followed instructions at: https://wiki.mozilla.org/Mobile/Fennec/Android#Testing
Blocks: 806369
Here's a patch which fixes the path issues that :gcp found earlier. Sorry, should have submitted this earlier.
Assignee: gbrown → wlachance
Attachment #676695 - Flags: review?(gbrown)
This fixes a minor error I found which confused me a bit when I was debugging stuff. Not that big a deal, but nice to have.
Attachment #676698 - Flags: review?(gbrown)
Comment on attachment 676695 [details] [diff] [review] Fix paths so remote xpcshell can find mozdevice Review of attachment 676695 [details] [diff] [review]: ----------------------------------------------------------------- This solves the devicemanager path problem.
Attachment #676695 - Flags: review?(gbrown) → review+
Comment on attachment 676698 [details] [diff] [review] Fix potential exception when variable is not defined Review of attachment 676698 [details] [diff] [review]: ----------------------------------------------------------------- Seems like a good idea!
Attachment #676698 - Flags: review?(gbrown) → review+
But I still cannot run xpcshell-tests-remote. With ADB: mozdev@mozdev-virtual-machine:~/src/objdir-native-droid$ make -C toolkit/mozapps/extensions/test xpcshell-tests-remote make: Entering directory `/home/mozdev/src/objdir-native-droid/toolkit/mozapps/extensions/test' /home/mozdev/src/objdir-native-droid/_virtualenv/bin/python -u /home/mozdev/src/config/pythonpath.py \ -I/home/mozdev/src/build \ -I/home/mozdev/src/build/mobile \ -I/home/mozdev/src/testing/mozbase/mozdevice/mozdevice \ /home/mozdev/src/testing/xpcshell/remotexpcshelltests.py \ --symbols-path=../../../../dist/crashreporter-symbols \ --build-info-json=../../../../mozinfo.json \ \ --dm_trans=adb \ --deviceIP= \ --objdir=../../../.. \ ../../../../_tests/xpcshell/toolkit/mozapps/extensions/test/xpcshell ../../../../_tests/xpcshell/toolkit/mozapps/extensions/test/xpcshell-unpack Android Debug Bridge version 1.0.29 'cp' not found, but 'dd' was found as a replacement 0+0 records in 0+0 records out 0 bytes transferred in 0.001 secs (0 bytes/sec) will execute commands via run-as org.mozilla.fennec_mozdev using APK: ../../../../dist/fennec-19.0a1.en-US.android-arm.apk Traceback (most recent call last): File "/home/mozdev/src/config/pythonpath.py", line 56, in <module> main(sys.argv[1:]) File "/home/mozdev/src/config/pythonpath.py", line 48, in main execfile(script, frozenglobals) File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 368, in <module> main() File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 359, in main xpcsh = XPCShellRemote(dm, options, args) File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 36, in __init__ self.setupUtilities() File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 81, in setupUtilities self.device.pushFile(local, self.remoteScriptsDir) File "/home/mozdev/src/testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py", line 169, in pushFile (localname, destname)) devicemanager.DMError: Attempted to push a file (/home/mozdev/src/testing/xpcshell/head.js) to a directory (/data/local/tests/xpcshell)! make: *** [xpcshell-tests-remote] Error 1 make: Leaving directory `/home/mozdev/src/objdir-native-droid/toolkit/mozapps/extensions/test' mozdev@mozdev-virtual-machine:~/src/objdir-native-droid$ With SUT: mozdev@mozdev-virtual-machine:~/src/objdir-native-droid$ make -C toolkit/mozapps/extensions/test xpcshell-tests-remote make: Entering directory `/home/mozdev/src/objdir-native-droid/toolkit/mozapps/extensions/test' /home/mozdev/src/objdir-native-droid/_virtualenv/bin/python -u /home/mozdev/src/config/pythonpath.py \ -I/home/mozdev/src/build \ -I/home/mozdev/src/build/mobile \ -I/home/mozdev/src/testing/mozbase/mozdevice/mozdevice \ /home/mozdev/src/testing/xpcshell/remotexpcshelltests.py \ --symbols-path=../../../../dist/crashreporter-symbols \ --build-info-json=../../../../mozinfo.json \ \ --dm_trans=sut \ --deviceIP=192.168.0.131 \ --objdir=../../../.. \ ../../../../_tests/xpcshell/toolkit/mozapps/extensions/test/xpcshell ../../../../_tests/xpcshell/toolkit/mozapps/extensions/test/xpcshell-unpack using APK: ../../../../dist/fennec-19.0a1.en-US.android-arm.apk Traceback (most recent call last): File "/home/mozdev/src/config/pythonpath.py", line 56, in <module> main(sys.argv[1:]) File "/home/mozdev/src/config/pythonpath.py", line 48, in main execfile(script, frozenglobals) File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 368, in <module> main() File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 359, in main xpcsh = XPCShellRemote(dm, options, args) File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 36, in __init__ self.setupUtilities() File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 81, in setupUtilities self.device.pushFile(local, self.remoteScriptsDir) File "/home/mozdev/src/testing/mozbase/mozdevice/mozdevice/devicemanagerSUT.py", line 341, in pushFile 'data': f.read() }]).strip() File "/home/mozdev/src/testing/mozbase/mozdevice/mozdevice/devicemanagerSUT.py", line 140, in _runCmds self._sendCmds(cmdlist, outputfile, timeout) File "/home/mozdev/src/testing/mozbase/mozdevice/mozdevice/devicemanagerSUT.py", line 122, in _sendCmds raise err devicemanager.DMError: Automation Error: Error processing command 'push /data/local/tests/xpcshell 27572'; err='Push failed!' make: *** [xpcshell-tests-remote] Error 1 make: Leaving directory `/home/mozdev/src/objdir-native-droid/toolkit/mozapps/extensions/test' mozdev@mozdev-virtual-machine:~/src/objdir-native-droid$
(In reply to Geoff Brown [:gbrown] from comment #12) > But I still cannot run xpcshell-tests-remote. > > With ADB: > ... > will execute commands via run-as org.mozilla.fennec_mozdev > using APK: ../../../../dist/fennec-19.0a1.en-US.android-arm.apk > Traceback (most recent call last): > File "/home/mozdev/src/config/pythonpath.py", line 56, in <module> > main(sys.argv[1:]) > File "/home/mozdev/src/config/pythonpath.py", line 48, in main > execfile(script, frozenglobals) > File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 368, > in <module> > main() > File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 359, > in main > xpcsh = XPCShellRemote(dm, options, args) > File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 36, > in __init__ > self.setupUtilities() > File "/home/mozdev/src/testing/xpcshell/remotexpcshelltests.py", line 81, > in setupUtilities > self.device.pushFile(local, self.remoteScriptsDir) > File > "/home/mozdev/src/testing/mozbase/mozdevice/mozdevice/devicemanagerADB.py", > line 169, in pushFile > (localname, destname)) > devicemanager.DMError: Attempted to push a file > (/home/mozdev/src/testing/xpcshell/head.js) to a directory > (/data/local/tests/xpcshell)! > make: *** [xpcshell-tests-remote] Error 1 > make: Leaving directory > `/home/mozdev/src/objdir-native-droid/toolkit/mozapps/extensions/test' > mozdev@mozdev-virtual-machine:~/src/objdir-native-droid$ I believe I recently modified devicemanagerADB to throw an exception in this case, as devicemanagerSUT doesn't support it and we want the two implementations to be interchangable (i.e. we don't want people writing dmADB-specific code). Hopefully this isn't too difficult to fix... if you want I can take a look at it later today/early tomorrow if you don't have time.
This seems to get me through the setup phase, at least with adb.
Attachment #676728 - Flags: review?(wlachance)
Comment on attachment 676728 [details] [diff] [review] use pushFile(file, dir/file) instead of pushFile(file, dir) Thanks for this. If it works for you, feel free to push it along with my patches. ># HG changeset patch ># Parent 12d85f27e4197974d97c9b1a3f57746435addc25 ># User Geoff Brown <gbrown@mozilla.com> >But 799863 - use dm.pushFile(file, dir/file) instead of dm.pushFile(file, dir); r=wlach > >diff --git a/testing/xpcshell/remotexpcshelltests.py b/testing/xpcshell/remotexpcshelltests.py >--- a/testing/xpcshell/remotexpcshelltests.py >+++ b/testing/xpcshell/remotexpcshelltests.py >@@ -73,38 +73,44 @@ class XPCShellRemote(xpcshell.XPCShellTe > if (not self.device.dirExists(remotePrefDir)): > self.device.mkDirs(self.remoteJoin(remotePrefDir, "extra")) > if (not self.device.dirExists(self.remoteScriptsDir)): > self.device.mkDir(self.remoteScriptsDir) > if (not self.device.dirExists(self.remoteComponentsDir)): > self.device.mkDir(self.remoteComponentsDir) > > local = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'head.js') >- self.device.pushFile(local, self.remoteScriptsDir) >+ remoteFile = self.remoteJoin(self.remoteScriptsDir, "head.js") >+ self.device.pushFile(local, remoteFile) > > localBin = os.path.join(self.options.objdir, "dist/bin") > if not os.path.exists(localBin): > localBin = os.path.join(self.options.objdir, "bin") > if not os.path.exists(localBin): > print >>sys.stderr, "Error: could not find bin in objdir" > sys.exit(1) > > local = os.path.join(localBin, "xpcshell") >- self.device.pushFile(local, self.remoteBinDir) >+ remoteFile = self.remoteJoin(self.remoteBinDir, "xpcshell") >+ self.device.pushFile(local, remoteFile) > > local = os.path.join(localBin, "components/httpd.js") >- self.device.pushFile(local, self.remoteComponentsDir) >+ remoteFile = self.remoteJoin(self.remoteComponentsDir, "httpd.js") >+ self.device.pushFile(local, remoteFile) > > local = os.path.join(localBin, "components/httpd.manifest") >- self.device.pushFile(local, self.remoteComponentsDir) >+ remoteFile = self.remoteJoin(self.remoteComponentsDir, "httpd.manifest") >+ self.device.pushFile(local, remoteFile) > > local = os.path.join(localBin, "components/test_necko.xpt") >- self.device.pushFile(local, self.remoteComponentsDir) >+ remoteFile = self.remoteJoin(self.remoteComponentsDir, "test_necko.xpt") >+ self.device.pushFile(local, remoteFile) > >- self.device.pushFile(self.options.localAPK, self.remoteBinDir) >+ remoteFile = self.remoteJoin(self.remoteBinDir, os.path.basename(self.options.localAPK)) >+ self.device.pushFile(self.options.localAPK, remoteFile) This looks like a fair bit of repeated code. Could we consolidate some of it into a member function?
Attachment #676728 - Flags: review?(wlachance) → review+
(In reply to William Lachance (:wlach) from comment #15) > Thanks for this. If it works for you, feel free to push it along with my > patches. Done: https://hg.mozilla.org/integration/mozilla-inbound/rev/c436e492f815 https://hg.mozilla.org/integration/mozilla-inbound/rev/8d1f8e125da1 https://hg.mozilla.org/integration/mozilla-inbound/rev/d2fd1f7c6c15 > This looks like a fair bit of repeated code. Could we consolidate some of it > into a member function? I opted not to pursue this; I have considered it before and it never ends up looking any better.
Depends on: 768491
Whiteboard: [leave open]
I am able to complete setup and run tests via adb now, but some problems remain: - setup takes a *very* long time to complete - a couple of hours! - most/all tests crash on exit: bug 768491 - "make check-one-remote" fails looking for devicemanager
A large number of files are copied to the device for xpcshell tests, which is very slow. It is enormously faster if busybox is installed on the device, since all the files can be transferred as a single zip.
(In reply to Geoff Brown [:gbrown] from comment #17) > I am able to complete setup and run tests via adb now, but some problems > remain: > > - setup takes a *very* long time to complete - a couple of hours! Yes, I noticed this too. We need to take advantage somehow of adb's ability to push an entire directory at a time here by modifying devicemanagerADB's pushDir implementation.
Setup via sut is much faster: just 13 minutes. But the tests themselves are not executed -- filed bug 807201.
Depends on: 807201
Depends on: 807841
We missed the "check-one-remote" make target earlier.
Attachment #677803 - Flags: review?(wlachance)
Comment on attachment 677803 [details] [diff] [review] Fix paths so check-one-remote can find mozdevice LGTM
Attachment #677803 - Flags: review?(wlachance) → review+
Setting a more descriptive title and unassigning me from the bug.
Assignee: wlachance → nobody
Summary: Cannot run xpcshell-tests-remote (no module named devicemanager) → xpcshell-tests-remote not working properly
Still working through the dependent bugs...
Assignee: nobody → gbrown
Blocks: 811793
Removing dependency on bug 768491 since it only affects tegras.
No longer depends on: 768491
xpcshell-tests-remote is healthy now: we can execute locally via adb or sut and test setup time is quite reasonable (I see less than 5 minutes of setup time with either adb or sut on a tegra).
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [leave open]
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: