https://wiki.mozilla.org/Mobile/Fennec/Android#Mochitests suggests that setting TEST_PATH to "content/smil/test" should run all the tests in that directory. Instead, it just displays a generic mochitest results page on Fennec, with Passed/Failed/Todo all set to 0. My recollection is that the last time I tried this (a month or two ago?) it worked, so I'm marking this as a regression.
after talking with dmose on irc, the root of this problem is that the profile is pushed to /data/local/tests where there is no permissions to access it from firefox. A single test will work just fine as we don't load the entire testrunner wrapper, likewise moving the testroot to /mnt/sdcard/tests instead of /data/local/tests will work in this situation.
I know we hit the opposite problem while testing bug 811411, in that we were pushing binaries to /mnt/sdcard/tests and failing when we couldn't chmod them +x. If we intend to support running tests on non-rooted devices we should make things work consistently in both cases: executables should always go to /data/local/tests, and files that need to be accessible from the app should go to /mnt/sdcard/tests. Alternately, we should add code to detect whether a device is rooted and ensure that it either does the right thing or errors with a clear message if that's impossible.
another problem is not all devices have /mnt/sdcard available. I would prefer to not support non rooted devices, but that is not reasonable as more people want to develop, debug, and test on personal devices. We could say that they must have a device with an available and working sdcard.
(In reply to Joel Maher (:jmaher) from comment #1) > after talking with dmose on irc, the root of this problem is that the > profile is pushed to /data/local/tests where there is no permissions to > access it from firefox. I think that worked previously. I wonder if this is because devicemanagerADB is no longer using run-as to push files in some circumstances.
The symptom (of the same problem) that happens when attempting to run a single test that requires SpecialPowers is that the test will attempt to execute, but fail when the in-browser JS code is unable to get a reference to the SpecialPowers object. jmaher suggested that that might be because in the single-test case, the same setup code isn't running.
I don't see this on a mozilla-central tip build from today when running tests for dom/apps/tests, which is a directory whose tests use SpecialPowers: 07-17 17:55 > TEST_PATH=dom/apps/tests make -C objdir-droid/ mochitest-remote ... 75 INFO Passed: 70 76 INFO Failed: 0 77 INFO Todo: 1 78 INFO SimpleTest FINISHED
Erm, never mind, I just realized the tests are using /mnt/sdcard/tests: Test root: /mnt/sdcard/tests
is this still a problem, I am sure we can fix/hack it to work for you in a reasonable amount of time.