Closed Bug 1445716 Opened 2 years ago Closed Last year

Run GeckoView junit tests in automation

Categories

(Firefox for Android :: Testing, defect, P1)

Unspecified
Android
defect

Tracking

()

RESOLVED FIXED
Firefox 61
Tracking Status
firefox-esr52 --- wontfix
firefox-esr60 --- wontfix
firefox59 --- wontfix
firefox60 --- wontfix
firefox61 --- fixed

People

(Reporter: snorp, Assigned: gbrown)

References

(Depends on 1 open bug)

Details

(Whiteboard: [geckoview:klar:p1])

Attachments

(7 files, 3 obsolete files)

We need to run the GeckoView JUnit tests in automation. These are in the org.mozilla.geckoview.test module. They are more-or-less standard Android tests and can be run with instructions similar to how it's described here: https://developer.android.com/studio/test/command-line.html#RunTestsGradle

These tests don't require any special environment, so anything should be suitable (emulator, hardware, whatever).
Assignee: nobody → gbrown
Priority: -- → P2
Summary: Run GeckoView tests in automation → Run GeckoView junit tests in automation
OS: Unspecified → Android
Whiteboard: [geckoview:klar]
Depends on: 1448188
P1 because we want to run tests before we ship Klar+GeckoView.
Priority: P2 → P1
Some initial results:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d70b35759dbe101ca76e989f674a512f40aeddc0

https://taskcluster-artifacts.net/dAHbU_H4QpGuavXgyFXqjw/0/public/logs/live_backing.log

task 2018-04-05T22:08:09.290Z] 22:08:09 CRITICAL -  ADBTimeoutError: args: adb -s emulator-5554 wait-for-device shell am instrument -w org.mozilla.geckoview.test/android.support.test.runner.AndroidJUnitRunner; echo rc=$?, exitcode: None, stdout:
[task 2018-04-05T22:08:09.291Z] 22:08:09     INFO -  org.mozilla.geckoview.test.ContentDelegateTest:
[task 2018-04-05T22:08:09.292Z] 22:08:09     INFO -  Error in titleChange(org.mozilla.geckoview.test.ContentDelegateTest):
[task 2018-04-05T22:08:09.292Z] 22:08:09     INFO -  java.lang.AssertionError: Timed out after 10000ms
[task 2018-04-05T22:08:09.293Z] 22:08:09     INFO -  	at org.junit.Assert.fail(Assert.java:88)
[task 2018-04-05T22:08:09.294Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891)
[task 2018-04-05T22:08:09.294Z] 22:08:09     INFO -  	at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.295Z] 22:08:09     INFO -  	at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.296Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915)
[task 2018-04-05T22:08:09.297Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788)
[task 2018-04-05T22:08:09.298Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729)
[task 2018-04-05T22:08:09.299Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842)
[task 2018-04-05T22:08:09.300Z] 22:08:09     INFO -  	at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44)
[task 2018-04-05T22:08:09.301Z] 22:08:09     INFO -  	at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719)
[task 2018-04-05T22:08:09.301Z] 22:08:09     INFO -  	at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.302Z] 22:08:09     INFO -  	at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.303Z] 22:08:09     INFO -  	at android.os.Looper.loop(Looper.java:137)
[task 2018-04-05T22:08:09.303Z] 22:08:09     INFO -  	at android.app.ActivityThread.main(ActivityThread.java:5103)
[task 2018-04-05T22:08:09.303Z] 22:08:09     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at dalvik.system.NativeStart.main(Native Method)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  Error in download(org.mozilla.geckoview.test.ContentDelegateTest):
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  java.lang.AssertionError: Timed out after 10000ms
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at org.junit.Assert.fail(Assert.java:88)
[task 2018-04-05T22:08:09.304Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$5.run(GeckoSessionTestRule.java:891)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.loopUntilIdle(GeckoSessionTestRule.java:915)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:788)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:729)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$3.evaluate(GeckoSessionTestRule.java:842)
[task 2018-04-05T22:08:09.305Z] 22:08:09     INFO -  	at android.support.test.internal.statement.UiThreadStatement$1.run(UiThreadStatement.java:44)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1719)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at android.os.Handler.handleCallback(Handler.java:730)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at android.os.Handler.dispatchMessage(Handler.java:92)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at android.os.Looper.loop(Looper.java:137)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at android.app.ActivityThread.main(ActivityThread.java:5103)
[task 2018-04-05T22:08:09.306Z] 22:08:09     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-04-05T22:08:09.307Z] 22:08:09     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-04-05T22:08:09.307Z] 22:08:09     INFO -  	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
[task 2018-04-05T22:08:09.307Z] 22:08:09     INFO -  	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
[task 2018-04-05T22:08:09.307Z] 22:08:09     INFO -  	at dalvik.system.NativeStart.main(Native Method)
[task 2018-04-05T22:08:09.308Z] 22:08:09     INFO -  org.mozilla.geckoview.test.GeckoSessionTestRuleTest:....................
[task 2018-04-05T22:08:09.308Z] 22:08:09     INFO -  Error in createOpenSession_withSettings(org.mozilla.geckoview.test.GeckoSessionTestRuleTest):
...


:jchen -- Are these "java.lang.AssertionError: Timed out after 10000ms" errors expected?
Flags: needinfo?(nchen)
I tried increasing the timeout to 60000ms, but that didn't help much:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=840428be63075924b3ee9a99f088d34300e2105e
Some of the failures are expected. Locally I get 4 failures under e10s and 3 under non-e10s. You should try with non-e10s by adding,

> mDefaultSettings.setBoolean(GeckoSessionSettings.USE_MULTIPROCESS, false);

at the end of `GeckoSessionTestRule()` near line 536 of GeckoSessionTestRule.java.

Also, FWIW, we will eventually need a mochitest server started up for some of the tests that require connection to a server (e.g. SSL tests).
Flags: needinfo?(nchen)
(In reply to Jim Chen [:jchen] [:darchons] from comment #4)
> Also, FWIW, we will eventually need a mochitest server started up for some
> of the tests that require connection to a server (e.g. SSL tests).

That should be easy, but...Recall that the mochitest harness creates a test profile with network proxies like "mochi.test", pushes that profile to the device and then runs Firefox for Android with -profile. Will we need similar facilities for these tests?
That should be pretty easy. "adb instrument" takes its arguments through "-e key value", so we can use that to pass in e10s flag, env vars, and Gecko command line arguments like -profile.
I should say our tests don't support these arguments right now, but if you come up with a list of arguments the test harness passes in, we can go ahead and add support for them.
Depends on: 1452239
OK, I'll create and push a profile for the network proxy support and we'll plan on using -profile to communicate the profile location. And run a mochitest-like server.
Another issue is the ability to skip troublesome tests.

I notice https://developer.mozilla.org/en-US/docs/Mozilla/Android-specific_test_suites#android-test talks about using @Ignore, but I could not get that to work (tried applying to the failing test in ContentDelegateTest.kt - build fails).
Hm that worked for me locally ("import org.junit.Ignore" and then add "@Ignore" in front of "@Test fun download()").

BTW related to bug 1452239, I think we want to eventually use the -r option for "adb instrument", which will continuously output raw results from individual test cases, and we can translate that into the treeherder format. I think that will provide a better treeherder experience (and less risk for timeout due to no output). This work can go in a follow-up bug though.
(In reply to Jim Chen [:jchen] [:darchons] from comment #10)
> Hm that worked for me locally ("import org.junit.Ignore" and then add
> "@Ignore" in front of "@Test fun download()").

Yes, indeed, that works. Must have had a typo before? Thanks!
Blocks: 1425322
I still need to finish off the harness, so cannot check-in yet, but let's look at the task definition now anyway.
Attachment #8967825 - Flags: review?(jmaher)
Comment on attachment 8967825 [details] [diff] [review]
add geckoview junit test task

Review of attachment 8967825 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with a few nits

::: taskcluster/ci/test/misc.yml
@@ +25,5 @@
> +    suite: junit
> +    treeherder-symbol: gv-junit
> +    instance-size: xlarge
> +    loopback-video: true
> +    e10s: false

isn't geckoview e10s?

::: testing/mozharness/mozharness/mozilla/testing/testbase.py
@@ +413,4 @@
>              'mochitest-plain-gpu': 'mochitest',
>              'mochitest-gl': 'mochitest',
>              'geckoview': 'mochitest',
> +            'junit': 'mochitest',

why does junit need the mochitest harness bits to download?
Attachment #8967825 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #15)
> > +    e10s: false
> 
> isn't geckoview e10s?

Good catch - thanks!

> ::: testing/mozharness/mozharness/mozilla/testing/testbase.py
> @@ +413,4 @@
> >              'mochitest-plain-gpu': 'mochitest',
> >              'mochitest-gl': 'mochitest',
> >              'geckoview': 'mochitest',
> > +            'junit': 'mochitest',
> 
> why does junit need the mochitest harness bits to download?

The harness does not reflect it yet, but will need to start a web server. I'm planning to leverage some mochitest harness code.
I've started a wiki page describing geckoview-junit tests at 

https://developer.mozilla.org/en-US/docs/Mozilla/Geckoview-Junit_Tests

Contributions welcome!
Depends on: 1454101
Depends on: 1454732
This simple mach command drives the runjunit.py test harness. All the normal Android test checks (device connected? app installed? host-utils found?) are in place. Implementation generally parallels existing code for other mochitest-related tests.

gbrown@mozpad:~/src$ ./mach geckoview-junit
No Android devices connected. Start an emulator? (Y/n) 
Starting emulator running Android 4.3...
Running the arm emulator; be sure to install an arm APK!
 0:00.31 adb INFO adbd running as root
 0:00.62 adb INFO su -c supported
 0:00.92 adb INFO su 0 supported
 0:01.43 adb INFO /system/bin/ls -a supported
 0:01.63 adb INFO Native cp support: True
 0:01.83 adb INFO Native chmod -R support: True
pk12util: PKCS12 IMPORT SUCCESSFUL
...
Attachment #8968714 - Flags: review?(jmaher)
gbrown@mozpad:~/src$ ./mach help geckoview-junit
usage: mach [global arguments] geckoview-junit [command arguments]

Run remote geckoview junit tests.

Global Arguments:
  -v, --verbose         Print verbose output.
  -l FILENAME, --log-file FILENAME
                        Filename to write log data to.
  --log-interval        Prefix log line with interval from last message rather
                        than relative time. Note that this is NOT execution
                        time if there are parallel operations.
  --log-no-times        Do not prefix log lines with times. By default, mach
                        will prefix each output line with the time since
                        command start.
  -h, --help            Show this help message.
  --debug-command       Start a Python debugger when command is dispatched.
  --settings FILENAME   Path to settings file.

Command Parameters:
  test_filters          Test filter(s): class and/or method names of test(s)
                        to run.

Command Arguments:
  -h, --help            show this help message and exit
  --appname APP         Test package name.
  --adbpath ADBPATH     Path to adb executable.
  --deviceSerial DEVICESERIAL
                        adb serial number of remote device.
  --remoteTestRoot REMOTETESTROOT
                        Remote directory to use as test root (eg.
                        /mnt/sdcard/tests or /data/local/tests).
  --disable-e10s        Disable multiprocess mode in test app.
  --max-time MAX_TIME   Max time in seconds to wait for tests (default 2400s).
  --runner RUNNER       Test runner name.
  --symbols-path SYMBOLSPATH
                        Path to directory containing breakpad symbols, or the
                        URL of a zip file containing symbols.
  --utility-path UTILITYPATH
                        Path to directory containing host utility programs.
  --certificate-path CERTPATH
                        Path to directory containing certificate store.
  --http-port HTTPPORT  Port of the web server for http traffic.
  --remote-webserver REMOTEWEBSERVER
                        IP address of the webserver.
  --ssl-port SSLPORT    Port of the web server for https traffic.

Output Logging:
  Each option represents a possible logging format and takes a filename to
  write that format to, or '-' to write to stdout. Some options are provided
  by the mozlog utility; see https://firefox-source-
  docs.mozilla.org/mozbase/mozlog.html for extended documentation.

  --log-unittest LOG_UNITTEST
                        Unittest style output (provided by mozlog)
...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f414ca0a30f0bc726baa181912b01f26f8b2ea68 demonstrates lots of good things: passing tests, treeherder-friendly logging, profile setup, crash reporting and symbols, and server setup.

I think we have everything we need to run these as a tier 1 job...except that there are several test failures and at least one crash due to a non-local network connection attempt.

:jchen -- Can you have a look at the test failures? Let me know if there is anything lacking in the test harness or environment.
Flags: needinfo?(nchen)
JUnitTestRunner inherits from MochitestDesktop simply to take advantage of mochitest's startServer/stopServer. Many options and class variables need to be set in order to use the mochitest servers: hence, server_init().

run_tests() starts the tests with something like 'adb shell am instrument -w -r ...' and uses an output callback to monitor the raw output from that command, line-by-line. Raw output is logged as structured logging "process output", and is also translated into structured logging test_start/test_end calls.
 
Otherwise, this is a pretty simple test harness.
Attachment #8967824 - Attachment is obsolete: true
Attachment #8968745 - Flags: review?(jmaher)
Comment on attachment 8968714 [details] [diff] [review]
add 'geckoview-junit' mach command

Review of attachment 8968714 [details] [diff] [review]:
-----------------------------------------------------------------

I have one question, will r+ if it is clarified.

::: testing/mochitest/mach_commands.py
@@ +275,5 @@
> +        with open(path, 'r') as fh:
> +            imp.load_module('mochitest', fh, path,
> +                            ('.py', 'r', imp.PY_SOURCE))
> +
> +        from runjunit import JunitArgumentParser

here you load_module 'mochitest', and above in run_geckoview_junit_test you load_module 'runjunit'.  My confusion is this line, where is runjunit defined as?
Attachment #8968714 - Flags: review?(jmaher) → review-
Comment on attachment 8968745 [details] [diff] [review]
add runjunit.py test harness

Review of attachment 8968745 [details] [diff] [review]:
-----------------------------------------------------------------

a few small nits

::: testing/mochitest/runjunit.py
@@ +165,5 @@
> +        env["MOZ_CRASHREPORTER_NO_REPORT"] = "1"
> +        env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1"
> +        env["XPCOM_DEBUG_BREAK"] = "stack"
> +        env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1"
> +        env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1"

on android don't we want to connect to a remote server?

@@ +217,5 @@
> +                if full_name == self.current_full_name:
> +                    if status == '0':
> +                        message = ''
> +                        status = 'PASS'
> +                        self.pass_count = self.pass_count + 1

why not use += 1?

@@ +221,5 @@
> +                        self.pass_count = self.pass_count + 1
> +                    else:
> +                        message = 'status %s' % status
> +                        status = 'FAIL'
> +                        self.fail_count = self.fail_count + 1

+= 1 ?

@@ +227,5 @@
> +                    self.test_started = False
> +                else:
> +                    if self.test_started:
> +                        # next test started without reporting previous status
> +                        self.fail_count = self.fail_count + 1

+= 1?

@@ +366,5 @@
> +                          help="Port of the web server for https traffic.")
> +        # Remaining arguments are test filters.
> +        self.add_argument("test_filters",
> +                          nargs="*",
> +                          help="Test filter(s): class and/or method names of test(s) to run.")

it seems like many of these cli options are already defined for remote testing or desktop testing- is there a way to share these command line parsers?  Maybe using mixins like we do in mozharness?
Attachment #8968745 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25)
> ::: testing/mochitest/runjunit.py
> @@ +165,5 @@
> > +        env["MOZ_CRASHREPORTER_NO_REPORT"] = "1"
> > +        env["MOZ_CRASHREPORTER_SHUTDOWN"] = "1"
> > +        env["XPCOM_DEBUG_BREAK"] = "stack"
> > +        env["DISABLE_UNSAFE_CPOW_WARNINGS"] = "1"
> > +        env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"] = "1"
> 
> on android don't we want to connect to a remote server?

I think we want the same as mochitests: Only allow white-listed proxied connections to our webserver. Otherwise tests may be dependent on external resources, leading to non-determinism and keeping us off of tier-1.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25)
> it seems like many of these cli options are already defined for remote
> testing or desktop testing- is there a way to share these command line
> parsers?  Maybe using mixins like we do in mozharness?

There are definitely lots of common options across test harnesses. One obstacle to sharing between say mochitest and reftest is the code location: Put it in mozbase somewhere? A mozoptions module??

At any rate, I think cleaning up the options is a separate bug.
In setup_junit_argument_parser (some of the earlier junit-related code involved in the mach command), if we load_module("runjunit.py"), it fails because the base class cannot be imported:

ImportError: cannot import name MochitestDesktop

  File "/home/gbrown/src/python/mach/mach/main.py", line 359, in run
    return self._run(argv)
  File "/home/gbrown/src/python/mach/mach/main.py", line 414, in _run
    args = parser.parse_args(argv)
  File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1942, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/gbrown/src/python/mach/mach/dispatcher.py", line 172, in __call__
    if handler.parser:
  File "/home/gbrown/src/python/mach/mach/decorators.py", line 76, in parser
    self._parser = self._parser()
  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 275, in setup_junit_argument_parser
    ('.py', 'r', imp.PY_SOURCE))
  File "/home/gbrown/objdirs/droid/_tests/testing/mochitest/runjunit.py", line 21, in <module>
    from runtests import MochitestDesktop, update_mozinfo

so we need to load_module("runtests.py") first. After that, load_module("runjunit.py") will succeed, but, afaict, that's not necessary: a simple import of runjunit works fine.

I've added a comment and simplified the imports accordingly.
Attachment #8968714 - Attachment is obsolete: true
Attachment #8969054 - Flags: review?(jmaher)
Comment on attachment 8969054 [details] [diff] [review]
add 'geckoview-junit' mach command

Review of attachment 8969054 [details] [diff] [review]:
-----------------------------------------------------------------

this looks much better, thanks.
Attachment #8969054 - Flags: review?(jmaher) → review+
I'll check in most of this, to make 'mach geckoview-junit' available, but won't enable junit in automation until we resolve the failing tests (comment 22).
Keywords: leave-open
Or maybe land as tier 2 until they're consistently green? I think most of the failures are due to a startup race that I'm looking at now.
Flags: needinfo?(nchen)
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5247cde8ef0c
Add runjunit.py, a test harness for junit tests on Android devices; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/05412fcc00a7
Add Android mach command 'geckoview-junit'; r=jmaher
(In reply to Jim Chen [:jchen] [:darchons] from comment #31)
> Or maybe land as tier 2 until they're consistently green? I think most of
> the failures are due to a startup race that I'm looking at now.

We could do that, but I hate to start running tests that consistently fail: The sheriffs will still file bugs and star the failures, and that can cause confusion (triagers trying to find ownership, without realizing its tier 2, etc).

It sounds like we are close; let's revisit in a few days.
Geoff, I filed bug 1456209 to address problems with the test framework. There are also some issues with the test harness that I found,

* In my try runs, the debug test run has a run time of ~90 minutes, so we need to increase the taskcluster timeout or split the debug run into two.

* We will have tests that make use of the mochitest host aliases (e.g. https://example.com), and those don't seem to work right now. I worked around it by adding a `locations` parameter at [1].

* The test proxy server and/or the SSL tunnel don't seem to be working. Loading https://example.com in a test always gave me a connection refused error, after fixing the `locations` problem above.

* The test runner uses status code -3 to mean "ignored" and -4 to mean "known fail", so we need to handle those. The full list of status codes is at [2].

* There are a lot of intermittent "missing test completion status" errors. I think there's something going on with the code that pipes adb output.

With these and bug 1456209 fixed, we should have some green runs!

[1] https://searchfox.org/mozilla-central/rev/fcd5cb3515d0e06592bba42e378f1b9518497e3d/testing/mochitest/runjunit.py#125

[2] https://android.googlesource.com/platform/frameworks/testing/+/master/support/src/android/support/test/internal/runner/listener/InstrumentationResultPrinter.java#62
Flags: needinfo?(gbrown)
Thanks Jim. I'm on it -- hope to have some solutions here soon.
Flags: needinfo?(gbrown)
I think it will be hard to chunk geckoview-junit because the harness does not have a manifest: tests are baked in to the apk and we run everything at once. Hopefully this will do.
Attachment #8970691 - Flags: review?(jmaher)
I don't know much about this stuff, but I've tried to make this parallel the code in runtests.py and it seems to work.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1b3d300585f132f71382f57c5cb696b35df7085
Attachment #8970692 - Flags: review?(jmaher)
Comment on attachment 8970690 [details] [diff] [review]
support skipped and expected-fail tests

Review of attachment 8970690 [details] [diff] [review]:
-----------------------------------------------------------------

nice and simple
Attachment #8970690 - Flags: review?(jmaher) → review+
Comment on attachment 8970691 [details] [diff] [review]
increase geckoview-junit max-run-time on debug

Review of attachment 8970691 [details] [diff] [review]:
-----------------------------------------------------------------

will these run faster on hardware?  If so then there is less concern.  I think we should find a way to make a manifest if possible- how can we skip  these tests and query what tests we have, etc... ?  Maybe we build different apk files for each chunk?

Either way, for now and the near future a single chunk sounds reasonable.
Attachment #8970691 - Flags: review?(jmaher) → review+
Comment on attachment 8970692 [details] [diff] [review]
support locations in geckoview-junit profile

Review of attachment 8970692 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mochitest/runjunit.py
@@ +182,5 @@
> +    @property
> +    def locations(self):
> +        if self._locations is not None:
> +            return self._locations
> +        locations_file = os.path.join(here, 'server-locations.txt')

do we copy server-locations.txt to the same directory as runjunit.py?  If so, then no concerns.  Ensure we handle both automation and mach.
Attachment #8970692 - Flags: review?(jmaher) → review+
Awesome work! The multipleLoads test is perma-fail right now (bug 1441059). The onNewSession_calledForNewWindow and onNewSession_doesNotAllowOpened tests are flaky (we're trying to fix that through bug 1456190). So ignoring those three, it's looking really good!

(In reply to Geoff Brown [:gbrown] from comment #38)
> Created attachment 8970691 [details] [diff] [review]
> increase geckoview-junit max-run-time on debug
> 
> I think it will be hard to chunk geckoview-junit because the harness does
> not have a manifest: tests are baked in to the apk and we run everything at
> once. Hopefully this will do.

The junit harness actually supports that natively [1], so might be worth looking into in the future.

[1] https://developer.android.com/training/testing/junit-runner.html#sharding-tests
(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #41)
> will these run faster on hardware?  If so then there is less concern.  I
> think we should find a way to make a manifest if possible- how can we skip 
> these tests and query what tests we have, etc... ?  Maybe we build different
> apk files for each chunk?

They likely will run faster on hardware, or an x86 emulator. There is native support for skipping tests (@Ignore in the source code) and selecting specific tests to run, so chunking was my only concern related to a manifest -- and with :jchen's tip (comment 43), I'm less concerned about that now. 

> Either way, for now and the near future a single chunk sounds reasonable.

I'll go ahead with this, and consider sharding another day.
(In reply to Joel Maher ( :jmaher - limited bugzilla access until May 1st) (UTC+2) from comment #42)
> do we copy server-locations.txt to the same directory as runjunit.py?  If
> so, then no concerns.  Ensure we handle both automation and mach.

It's just like mochitest/runtests.py and works fine locally.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1b29fa5bb802
Process more geckoview-junit status codes; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056
Add new Android test task geckoview-junit, but do not run it yet; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/ec40107c1c13
Increase timeout for geckoview-junit on android debug; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/fe6ddc27db72
Support profile locations in geckoview-junit; r=jmaher
I included the task definition in this check-in, but disabled it. To test on try, we'll still need to remove the comment at:

https://hg.mozilla.org/integration/mozilla-inbound/rev/dbcb5f8ea056f2307b867e9ceafa4772b91c2a00#l3.12

Still to do:
 - last few failing tests (comment 43)
 - possible lost output / missing status code problem
Until those tests get fixed in their bugs, feel free to @Ignore them for now.
[geckoview:klar:p1] because running these tests is a Klar+GeckoView blocker.
Whiteboard: [geckoview:klar] → [geckoview:klar:p1]
Depends on: 1457662
Attached patch skip failing tests (obsolete) — Splinter Review
As suggested, @Ignore failing tests to get a green run. We can start running geckoview-junit on trunk with this change and bug 1457662.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=30d3cf2e1cbf8cc0ea25c628b271022dde716591
Attachment #8972008 - Flags: review?(nchen)
Comment on attachment 8972008 [details] [diff] [review]
skip failing tests

There are still some debug-only failures...I'll revise a bit...
Attachment #8972008 - Flags: review?(nchen)
Depends on: 1457971
Attachment #8972117 - Flags: review?(nchen) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e131aaef5d89
Run new Android test task, geckoview-junit; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/a340a6ab8489
Skip remaining failures in geckoview-junit tests; r=jchen
Keywords: leave-open
https://hg.mozilla.org/mozilla-central/rev/e131aaef5d89
https://hg.mozilla.org/mozilla-central/rev/a340a6ab8489
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → Firefox 61
You need to log in before you can comment on or make changes to this bug.