Closed
Bug 1445716
Opened 7 years ago
Closed 7 years ago
Run GeckoView junit tests in automation
Categories
(Firefox for Android Graveyard :: Testing, defect, P1)
Tracking
(firefox-esr52 wontfix, firefox-esr60 wontfix, firefox59 wontfix, firefox60 wontfix, firefox61 fixed)
RESOLVED
FIXED
Firefox 61
People
(Reporter: snorp, Assigned: gbrown)
References
Details
(Whiteboard: [geckoview:klar:p1])
Attachments
(7 files, 3 obsolete files)
3.78 KB,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
17.33 KB,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
3.69 KB,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
3.00 KB,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
621 bytes,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
1.50 KB,
patch
|
jmaher
:
review+
|
Details | Diff | Splinter Review |
4.33 KB,
patch
|
jchen
:
review+
|
Details | Diff | Splinter Review |
We need to run the GeckoView JUnit tests in automation. These are in the org.mozilla.geckoview.test module. They are more-or-less standard Android tests and can be run with instructions similar to how it's described here: https://developer.android.com/studio/test/command-line.html#RunTestsGradle
These tests don't require any special environment, so anything should be suitable (emulator, hardware, whatever).
Assignee | ||
Updated•7 years ago
|
Assignee: nobody → gbrown
Priority: -- → P2
Assignee | ||
Updated•7 years ago
|
Summary: Run GeckoView tests in automation → Run GeckoView junit tests in automation
Updated•7 years ago
|
status-firefox59:
--- → wontfix
status-firefox60:
--- → wontfix
status-firefox61:
--- → affected
OS: Unspecified → Android
Whiteboard: [geckoview:klar]
Comment 1•7 years ago
|
||
P1 because we want to run tests before we ship Klar+GeckoView.
status-firefox-esr52:
--- → wontfix
Priority: P2 → P1
Assignee | ||
Comment 2•7 years ago
|
||
Some initial results:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d70b35759dbe101ca76e989f674a512f40aeddc0
https://taskcluster-artifacts.net/dAHbU_H4QpGuavXgyFXqjw/0/public/logs/live_backing.log
task 2018-04-05T22:08:09.290Z] 22:08:09 CRITICAL - ADBTimeoutError: args: adb -s emulator-5554 wait-for-device shell am instrument -w org.mozilla.geckoview.test/android.support.test.runner.AndroidJUnitRunner; echo rc=$?, exitcode: None, stdout:
[task 2018-04-05T22:08:09.291Z] 22:08:09 INFO - org.mozilla.geckoview.test.ContentDelegateTest:
[task 2018-04-05T22:08:09.292Z] 22:08:09 INFO - Error in titleChange(org.mozilla.geckoview.test.ContentDelegateTest):
[task 2018-04-05T22:08:09.292Z] 22:08:09 INFO - java.lang.AssertionError: Timed out after 10000ms
[task 2018-04-05T22:08:09.293Z] 22:08:09 INFO - at org.junit.Assert.fail(Assert.java:88)
[task 2018-04-05T22:08:09.294Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891)
[task 2018-04-05T22:08:09.294Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.295Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.296Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915)
[task 2018-04-05T22:08:09.297Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788)
[task 2018-04-05T22:08:09.298Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729)
[task 2018-04-05T22:08:09.299Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842)
[task 2018-04-05T22:08:09.300Z] 22:08:09 INFO - at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44)
[task 2018-04-05T22:08:09.301Z] 22:08:09 INFO - at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719)
[task 2018-04-05T22:08:09.301Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.302Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at android.os.Looper.loop(Looper.java:137)
[task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at android.app.ActivityThread.main(ActivityThread.java:5103)
[task 2018-04-05T22:08:09.303Z] 22:08:09 INFO - at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at dalvik.system.NativeStart.main(Native Method)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - Error in download(org.mozilla.geckoview.test.ContentDelegateTest):
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - java.lang.AssertionError: Timed out after 10000ms
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at org.junit.Assert.fail(Assert.java:88)
[task 2018-04-05T22:08:09.304Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842)
[task 2018-04-05T22:08:09.305Z] 22:08:09 INFO - at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.os.Looper.loop(Looper.java:137)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at android.app.ActivityThread.main(ActivityThread.java:5103)
[task 2018-04-05T22:08:09.306Z] 22:08:09 INFO - at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
[task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
[task 2018-04-05T22:08:09.307Z] 22:08:09 INFO - at dalvik.system.NativeStart.main(Native Method)
[task 2018-04-05T22:08:09.308Z] 22:08:09 INFO - org.mozilla.geckoview.test.GeckoSessionTestRuleTest:....................
[task 2018-04-05T22:08:09.308Z] 22:08:09 INFO - Error in createOpenSession_withSettings(org.mozilla.geckoview.test.GeckoSessionTestRuleTest):
...
:jchen -- Are these "java.lang.AssertionError: Timed out after 10000ms" errors expected?
Flags: needinfo?(nchen)
Assignee | ||
Comment 3•7 years ago
|
||
I tried increasing the timeout to 60000ms, but that didn't help much:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=840428be63075924b3ee9a99f088d34300e2105e
Comment 4•7 years ago
|
||
Some of the failures are expected. Locally I get 4 failures under e10s and 3 under non-e10s. You should try with non-e10s by adding,
> mDefaultSettings.setBoolean(GeckoSessionSettings.USE_MULTIPROCESS, false);
at the end of `GeckoSessionTestRule()` near line 536 of GeckoSessionTestRule.java.
Also, FWIW, we will eventually need a mochitest server started up for some of the tests that require connection to a server (e.g. SSL tests).
Flags: needinfo?(nchen)
Assignee | ||
Comment 5•7 years ago
|
||
(In reply to Jim Chen [:jchen] [:darchons] from comment #4)
> Also, FWIW, we will eventually need a mochitest server started up for some
> of the tests that require connection to a server (e.g. SSL tests).
That should be easy, but...Recall that the mochitest harness creates a test profile with network proxies like "mochi.test", pushes that profile to the device and then runs Firefox for Android with -profile. Will we need similar facilities for these tests?
Comment 6•7 years ago
|
||
That should be pretty easy. "adb instrument" takes its arguments through "-e key value", so we can use that to pass in e10s flag, env vars, and Gecko command line arguments like -profile.
Comment 7•7 years ago
|
||
I should say our tests don't support these arguments right now, but if you come up with a list of arguments the test harness passes in, we can go ahead and add support for them.
Assignee | ||
Comment 8•7 years ago
|
||
OK, I'll create and push a profile for the network proxy support and we'll plan on using -profile to communicate the profile location. And run a mochitest-like server.
Assignee | ||
Comment 9•7 years ago
|
||
Another issue is the ability to skip troublesome tests.
I notice https://developer.mozilla.org/en-US/docs/Mozilla/Android-specific_test_suites#android-test talks about using @Ignore, but I could not get that to work (tried applying to the failing test in ContentDelegateTest.kt - build fails).
Comment 10•7 years ago
|
||
Hm that worked for me locally ("import org.junit.Ignore" and then add "@Ignore" in front of "@Test fun download()").
BTW related to bug 1452239, I think we want to eventually use the -r option for "adb instrument", which will continuously output raw results from individual test cases, and we can translate that into the treeherder format. I think that will provide a better treeherder experience (and less risk for timeout due to no output). This work can go in a follow-up bug though.
Assignee | ||
Comment 11•7 years ago
|
||
(In reply to Jim Chen [:jchen] [:darchons] from comment #10)
> Hm that worked for me locally ("import org.junit.Ignore" and then add
> "@Ignore" in front of "@Test fun download()").
Yes, indeed, that works. Must have had a typo before? Thanks!
Assignee | ||
Comment 12•7 years ago
|
||
With -r and some minor logging improvements:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9a6b523d852c6d291451584de5e51b9e54f58c6e
Assignee | ||
Comment 13•7 years ago
|
||
Assignee | ||
Comment 14•7 years ago
|
||
I still need to finish off the harness, so cannot check-in yet, but let's look at the task definition now anyway.
Attachment #8967825 -
Flags: review?(jmaher)
Comment 15•7 years ago
|
||
Comment on attachment 8967825 [details] [diff] [review]
add geckoview junit test task
Review of attachment 8967825 [details] [diff] [review]:
-----------------------------------------------------------------
r=me with a few nits
::: taskcluster/ci/test/misc.yml
@@ +25,5 @@
> + suite: junit
> + treeherder-symbol: gv-junit
> + instance-size: xlarge
> + loopback-video: true
> + e10s: false
isn't geckoview e10s?
::: testing/mozharness/mozharness/mozilla/testing/testbase.py
@@ +413,4 @@
> 'mochitest-plain-gpu': 'mochitest',
> 'mochitest-gl': 'mochitest',
> 'geckoview': 'mochitest',
> + 'junit': 'mochitest',
why does junit need the mochitest harness bits to download?
Attachment #8967825 -
Flags: review?(jmaher) → review+
Assignee | ||
Comment 16•7 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #15)
> > + e10s: false
>
> isn't geckoview e10s?
Good catch - thanks!
> ::: testing/mozharness/mozharness/mozilla/testing/testbase.py
> @@ +413,4 @@
> > 'mochitest-plain-gpu': 'mochitest',
> > 'mochitest-gl': 'mochitest',
> > 'geckoview': 'mochitest',
> > + 'junit': 'mochitest',
>
> why does junit need the mochitest harness bits to download?
The harness does not reflect it yet, but will need to start a web server. I'm planning to leverage some mochitest harness code.
Assignee | ||
Comment 17•7 years ago
|
||
I've started a wiki page describing geckoview-junit tests at
https://developer.mozilla.org/en-US/docs/Mozilla/Geckoview-Junit_Tests
Contributions welcome!
Assignee | ||
Comment 18•7 years ago
|
||
Assignee | ||
Comment 19•7 years ago
|
||
Now with e10s, profile, crash reporting and symbols:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=80f86d7124fe506c6932a2164734de3b65b6961f
Assignee | ||
Comment 20•7 years ago
|
||
This simple mach command drives the runjunit.py test harness. All the normal Android test checks (device connected? app installed? host-utils found?) are in place. Implementation generally parallels existing code for other mochitest-related tests.
gbrown@mozpad:~/src$ ./mach geckoview-junit
No Android devices connected. Start an emulator? (Y/n)
Starting emulator running Android 4.3...
Running the arm emulator; be sure to install an arm APK!
0:00.31 adb INFO adbd running as root
0:00.62 adb INFO su -c supported
0:00.92 adb INFO su 0 supported
0:01.43 adb INFO /system/bin/ls -a supported
0:01.63 adb INFO Native cp support: True
0:01.83 adb INFO Native chmod -R support: True
pk12util: PKCS12 IMPORT SUCCESSFUL
...
Attachment #8968714 -
Flags: review?(jmaher)
Assignee | ||
Comment 21•7 years ago
|
||
gbrown@mozpad:~/src$ ./mach help geckoview-junit
usage: mach [global arguments] geckoview-junit [command arguments]
Run remote geckoview junit tests.
Global Arguments:
-v, --verbose Print verbose output.
-l FILENAME, --log-file FILENAME
Filename to write log data to.
--log-interval Prefix log line with interval from last message rather
than relative time. Note that this is NOT execution
time if there are parallel operations.
--log-no-times Do not prefix log lines with times. By default, mach
will prefix each output line with the time since
command start.
-h, --help Show this help message.
--debug-command Start a Python debugger when command is dispatched.
--settings FILENAME Path to settings file.
Command Parameters:
test_filters Test filter(s): class and/or method names of test(s)
to run.
Command Arguments:
-h, --help show this help message and exit
--appname APP Test package name.
--adbpath ADBPATH Path to adb executable.
--deviceSerial DEVICESERIAL
adb serial number of remote device.
--remoteTestRoot REMOTETESTROOT
Remote directory to use as test root (eg.
/mnt/sdcard/tests or /data/local/tests).
--disable-e10s Disable multiprocess mode in test app.
--max-time MAX_TIME Max time in seconds to wait for tests (default 2400s).
--runner RUNNER Test runner name.
--symbols-path SYMBOLSPATH
Path to directory containing breakpad symbols, or the
URL of a zip file containing symbols.
--utility-path UTILITYPATH
Path to directory containing host utility programs.
--certificate-path CERTPATH
Path to directory containing certificate store.
--http-port HTTPPORT Port of the web server for http traffic.
--remote-webserver REMOTEWEBSERVER
IP address of the webserver.
--ssl-port SSLPORT Port of the web server for https traffic.
Output Logging:
Each option represents a possible logging format and takes a filename to
write that format to, or '-' to write to stdout. Some options are provided
by the mozlog utility; see https://firefox-source-
docs.mozilla.org/mozbase/mozlog.html for extended documentation.
--log-unittest LOG_UNITTEST
Unittest style output (provided by mozlog)
...
Assignee | ||
Comment 22•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f414ca0a30f0bc726baa181912b01f26f8b2ea68 demonstrates lots of good things: passing tests, treeherder-friendly logging, profile setup, crash reporting and symbols, and server setup.
I think we have everything we need to run these as a tier 1 job...except that there are several test failures and at least one crash due to a non-local network connection attempt.
:jchen -- Can you have a look at the test failures? Let me know if there is anything lacking in the test harness or environment.
Flags: needinfo?(nchen)
Assignee | ||
Comment 23•7 years ago
|
||
JUnitTestRunner inherits from MochitestDesktop simply to take advantage of mochitest's startServer/stopServer. Many options and class variables need to be set in order to use the mochitest servers: hence, server_init().
run_tests() starts the tests with something like 'adb shell am instrument -w -r ...' and uses an output callback to monitor the raw output from that command, line-by-line. Raw output is logged as structured logging "process output", and is also translated into structured logging test_start/test_end calls.
Otherwise, this is a pretty simple test harness.
Attachment #8967824 -
Attachment is obsolete: true
Attachment #8968745 -
Flags: review?(jmaher)
Comment 24•7 years ago
|
||
Comment on attachment 8968714 [details] [diff] [review]
add 'geckoview-junit' mach command
Review of attachment 8968714 [details] [diff] [review]:
-----------------------------------------------------------------
I have one question, will r+ if it is clarified.
::: testing/mochitest/mach_commands.py
@@ +275,5 @@
> + with open(path, 'r') as fh:
> + imp.load_module('mochitest', fh, path,
> + ('.py', 'r', imp.PY_SOURCE))
> +
> + from runjunit import JunitArgumentParser
here you load_module 'mochitest', and above in run_geckoview_junit_test you load_module 'runjunit'. My confusion is this line, where is runjunit defined as?
Attachment #8968714 -
Flags: review?(jmaher) → review-
Comment 25•7 years ago
|
||
Comment on attachment 8968745 [details] [diff] [review]
add runjunit.py test harness
Review of attachment 8968745 [details] [diff] [review]:
-----------------------------------------------------------------
a few small nits
::: testing/mochitest/runjunit.py
@@ +165,5 @@
> + env["MOZ_CRASHREPORTER_NO_REPORT"] = "1"
> + env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1"
> + env["XPCOM_DEBUG_BREAK"] = "stack"
> + env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1"
> + env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1"
on android don't we want to connect to a remote server?
@@ +217,5 @@
> + if full_name == self.current_full_name:
> + if status == '0':
> + message = ''
> + status = 'PASS'
> + self.pass_count = self.pass_count + 1
why not use += 1?
@@ +221,5 @@
> + self.pass_count = self.pass_count + 1
> + else:
> + message = 'status %s' % status
> + status = 'FAIL'
> + self.fail_count = self.fail_count + 1
+= 1 ?
@@ +227,5 @@
> + self.test_started = False
> + else:
> + if self.test_started:
> + # next test started without reporting previous status
> + self.fail_count = self.fail_count + 1
+= 1?
@@ +366,5 @@
> + help="Port of the web server for https traffic.")
> + # Remaining arguments are test filters.
> + self.add_argument("test_filters",
> + nargs="*",
> + help="Test filter(s): class and/or method names of test(s) to run.")
it seems like many of these cli options are already defined for remote testing or desktop testing- is there a way to share these command line parsers? Maybe using mixins like we do in mozharness?
Attachment #8968745 -
Flags: review?(jmaher) → review+
Assignee | ||
Comment 26•7 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25)
> ::: testing/mochitest/runjunit.py
> @@ +165,5 @@
> > + env["MOZ_CRASHREPORTER_NO_REPORT"] = "1"
> > + env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1"
> > + env["XPCOM_DEBUG_BREAK"] = "stack"
> > + env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1"
> > + env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1"
>
> on android don't we want to connect to a remote server?
I think we want the same as mochitests: Only allow white-listed proxied connections to our webserver. Otherwise tests may be dependent on external resources, leading to non-determinism and keeping us off of tier-1.
Assignee | ||
Comment 27•7 years ago
|
||
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25)
> it seems like many of these cli options are already defined for remote
> testing or desktop testing- is there a way to share these command line
> parsers? Maybe using mixins like we do in mozharness?
There are definitely lots of common options across test harnesses. One obstacle to sharing between say mochitest and reftest is the code location: Put it in mozbase somewhere? A mozoptions module??
At any rate, I think cleaning up the options is a separate bug.
Assignee | ||
Comment 28•7 years ago
|
||
In setup_junit_argument_parser (some of the earlier junit-related code involved in the mach command), if we load_module("runjunit.py"), it fails because the base class cannot be imported:
ImportError: cannot import name MochitestDesktop
File "/home/gbrown/src/python/mach/mach/main.py", line 359, in run
return self._run(argv)
File "/home/gbrown/src/python/mach/mach/main.py", line 414, in _run
args = parser.parse_args(argv)
File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
stop_index = consume_positionals(start_index)
File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
take_action(action, args)
File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
action(self, namespace, argument_values, option_string)
File "/home/gbrown/src/python/mach/mach/dispatcher.py", line 172, in __call__
if handler.parser:
File "/home/gbrown/src/python/mach/mach/decorators.py", line 76, in parser
self._parser = self._parser()
File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 275, in setup_junit_argument_parser
('.py', 'r', imp.PY_SOURCE))
File "/home/gbrown/objdirs/droid/_tests/testing/mochitest/runjunit.py", line 21, in <module>
from runtests import MochitestDesktop, update_mozinfo
so we need to load_module("runtests.py") first. After that, load_module("runjunit.py") will succeed, but, afaict, that's not necessary: a simple import of runjunit works fine.
I've added a comment and simplified the imports accordingly.
Attachment #8968714 -
Attachment is obsolete: true
Attachment #8969054 -
Flags: review?(jmaher)
Comment 29•7 years ago
|
||
Comment on attachment 8969054 [details] [diff] [review]
add 'geckoview-junit' mach command
Review of attachment 8969054 [details] [diff] [review]:
-----------------------------------------------------------------
this looks much better, thanks.
Attachment #8969054 -
Flags: review?(jmaher) → review+
Assignee | ||
Comment 30•7 years ago
|
||
I'll check in most of this, to make 'mach geckoview-junit' available, but won't enable junit in automation until we resolve the failing tests (comment 22).
Keywords: leave-open
Comment 31•7 years ago
|
||
Or maybe land as tier 2 until they're consistently green? I think most of the failures are due to a startup race that I'm looking at now.
Flags: needinfo?(nchen)
Comment 32•7 years ago
|
||
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5247cde8ef0c
Add runjunit.py, a test harness for junit tests on Android devices; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/05412fcc00a7
Add Android mach command 'geckoview-junit'; r=jmaher
Assignee | ||
Comment 33•7 years ago
|
||
(In reply to Jim Chen [:jchen] [:darchons] from comment #31)
> Or maybe land as tier 2 until they're consistently green? I think most of
> the failures are due to a startup race that I'm looking at now.
We could do that, but I hate to start running tests that consistently fail: The sheriffs will still file bugs and star the failures, and that can cause confusion (triagers trying to find ownership, without realizing its tier 2, etc).
It sounds like we are close; let's revisit in a few days.
Comment 34•7 years ago
|
||
bugherder |
Comment 35•7 years ago
|
||
Geoff, I filed bug 1456209 to address problems with the test framework. There are also some issues with the test harness that I found,
* In my try runs, the debug test run has a run time of ~90 minutes, so we need to increase the taskcluster timeout or split the debug run into two.
* We will have tests that make use of the mochitest host aliases (e.g. https://example.com), and those don't seem to work right now. I worked around it by adding a `locations` parameter at [1].
* The test proxy server and/or the SSL tunnel don't seem to be working. Loading https://example.com in a test always gave me a connection refused error, after fixing the `locations` problem above.
* The test runner uses status code -3 to mean "ignored" and -4 to mean "known fail", so we need to handle those. The full list of status codes is at [2].
* There are a lot of intermittent "missing test completion status" errors. I think there's something going on with the code that pipes adb output.
With these and bug 1456209 fixed, we should have some green runs!
[1] https://searchfox.org/mozilla-central/rev/fcd5cb3515d0e06592bba42e378f1b9518497e3d/testing/mochitest/runjunit.py#125
[2] https://android.googlesource.com/platform/frameworks/testing/+/master/support/src/android/support/test/internal/runner/listener/InstrumentationResultPrinter.java#62
Flags: needinfo?(gbrown)
Assignee | ||
Comment 36•7 years ago
|
||
Thanks Jim. I'm on it -- hope to have some solutions here soon.
Flags: needinfo?(gbrown)
Assignee | ||
Comment 37•7 years ago
|
||
Attachment #8970690 -
Flags: review?(jmaher)
Assignee | ||
Comment 38•7 years ago
|
||
I think it will be hard to chunk geckoview-junit because the harness does not have a manifest: tests are baked in to the apk and we run everything at once. Hopefully this will do.
Attachment #8970691 -
Flags: review?(jmaher)
Assignee | ||
Comment 39•7 years ago
|
||
I don't know much about this stuff, but I've tried to make this parallel the code in runtests.py and it seems to work.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1b3d300585f132f71382f57c5cb696b35df7085
Attachment #8970692 -
Flags: review?(jmaher)
Comment 40•7 years ago
|
||
Comment on attachment 8970690 [details] [diff] [review]
support skipped and expected-fail tests
Review of attachment 8970690 [details] [diff] [review]:
-----------------------------------------------------------------
nice and simple
Attachment #8970690 -
Flags: review?(jmaher) → review+
Comment 41•7 years ago
|
||
Comment on attachment 8970691 [details] [diff] [review]
increase geckoview-junit max-run-time on debug
Review of attachment 8970691 [details] [diff] [review]:
-----------------------------------------------------------------
will these run faster on hardware? If so then there is less concern. I think we should find a way to make a manifest if possible- how can we skip these tests and query what tests we have, etc... ? Maybe we build different apk files for each chunk?
Either way, for now and the near future a single chunk sounds reasonable.
Attachment #8970691 -
Flags: review?(jmaher) → review+
Comment 42•7 years ago
|
||
Comment on attachment 8970692 [details] [diff] [review]
support locations in geckoview-junit profile
Review of attachment 8970692 [details] [diff] [review]:
-----------------------------------------------------------------
::: testing/mochitest/runjunit.py
@@ +182,5 @@
> + @property
> + def locations(self):
> + if self._locations is not None:
> + return self._locations
> + locations_file = os.path.join(here, 'server-locations.txt')
do we copy server-locations.txt to the same directory as runjunit.py? If so, then no concerns. Ensure we handle both automation and mach.
Attachment #8970692 -
Flags: review?(jmaher) → review+
Comment 43•7 years ago
|
||
Awesome work! The multipleLoads test is perma-fail right now (bug 1441059). The onNewSession_calledForNewWindow and onNewSession_doesNotAllowOpened tests are flaky (we're trying to fix that through bug 1456190). So ignoring those three, it's looking really good!
(In reply to Geoff Brown [:gbrown] from comment #38)
> Created attachment 8970691 [details] [diff] [review]
> increase geckoview-junit max-run-time on debug
>
> I think it will be hard to chunk geckoview-junit because the harness does
> not have a manifest: tests are baked in to the apk and we run everything at
> once. Hopefully this will do.
The junit harness actually supports that natively [1], so might be worth looking into in the future.
[1] https://developer.android.com/training/testing/junit-runner.html#sharding-tests
Assignee | ||
Comment 44•7 years ago
|
||
(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #41)
> will these run faster on hardware? If so then there is less concern. I
> think we should find a way to make a manifest if possible- how can we skip
> these tests and query what tests we have, etc... ? Maybe we build different
> apk files for each chunk?
They likely will run faster on hardware, or an x86 emulator. There is native support for skipping tests (@Ignore in the source code) and selecting specific tests to run, so chunking was my only concern related to a manifest -- and with :jchen's tip (comment 43), I'm less concerned about that now.
> Either way, for now and the near future a single chunk sounds reasonable.
I'll go ahead with this, and consider sharding another day.
Assignee | ||
Comment 45•7 years ago
|
||
(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #42)
> do we copy server-locations.txt to the same directory as runjunit.py? If
> so, then no concerns. Ensure we handle both automation and mach.
It's just like mochitest/runtests.py and works fine locally.
Comment 46•7 years ago
|
||
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1b29fa5bb802
Process more geckoview-junit status codes; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056
Add new Android test task geckoview-junit, but do not run it yet; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/ec40107c1c13
Increase timeout for geckoview-junit on android debug; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/fe6ddc27db72
Support profile locations in geckoview-junit; r=jmaher
Assignee | ||
Comment 47•7 years ago
|
||
I included the task definition in this check-in, but disabled it. To test on try, we'll still need to remove the comment at:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056f2307b867e9ceafa4772b91c2a00#l3.12
Still to do:
- last few failing tests (comment 43)
- possible lost output / missing status code problem
Comment 48•7 years ago
|
||
Until those tests get fixed in their bugs, feel free to @Ignore them for now.
Comment 49•7 years ago
|
||
bugherder |
Comment 50•7 years ago
|
||
[geckoview:klar:p1] because running these tests is a Klar+GeckoView blocker.
status-firefox-esr60:
--- → wontfix
Whiteboard: [geckoview:klar] → [geckoview:klar:p1]
Assignee | ||
Comment 51•7 years ago
|
||
As suggested, @Ignore failing tests to get a green run. We can start running geckoview-junit on trunk with this change and bug 1457662.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=30d3cf2e1cbf8cc0ea25c628b271022dde716591
Attachment #8972008 -
Flags: review?(nchen)
Assignee | ||
Comment 52•7 years ago
|
||
Comment on attachment 8972008 [details] [diff] [review]
skip failing tests
There are still some debug-only failures...I'll revise a bit...
Attachment #8972008 -
Flags: review?(nchen)
Assignee | ||
Comment 53•7 years ago
|
||
Looking good now:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1c1d14b5362caf408d5fdaadae126790222a2c3
Attachment #8972008 -
Attachment is obsolete: true
Attachment #8972117 -
Flags: review?(nchen)
Updated•7 years ago
|
Attachment #8972117 -
Flags: review?(nchen) → review+
Comment 54•7 years ago
|
||
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e131aaef5d89
Run new Android test task, geckoview-junit; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/a340a6ab8489
Skip remaining failures in geckoview-junit tests; r=jchen
Assignee | ||
Updated•7 years ago
|
Keywords: leave-open
Updated•7 years ago
|
Comment 55•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/e131aaef5d89
https://hg.mozilla.org/mozilla-central/rev/a340a6ab8489
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 61
Updated•4 years ago
|
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•