1496627 - Get web-platform-tests running locally on an android emulator

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1496627 - Make sure emulator is running

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1496627 - Run the emulator automatically r?gbrown

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 9014665 [details]
Bug 1496627 - Make sure emulator is running

Messed up arcanist or something so a new request appeared...

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1496627 - Move the import into the function where it is needed

Depends on D8260

Comment 5

[Geoff Brown [:gbrown]]

:kwierso sent me a logcat including:

10-10 22:34:08.953  2793  2808 E GeckoProfile: Error getting profile dir
10-10 22:34:08.953  2793  2808 E GeckoProfile: java.io.IOException: Unable to create profile.
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at org.mozilla.gecko.GeckoProfile.createProfileDir(GeckoProfile.java:906)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at org.mozilla.gecko.GeckoProfile.forceCreateLocked(GeckoProfile.java:416)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at org.mozilla.gecko.GeckoProfile.getDir(GeckoProfile.java:396)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at org.mozilla.gecko.GeckoApp$10.run(GeckoApp.java:1238)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at android.os.Handler.handleCallback(Handler.java:751)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at android.os.Handler.dispatchMessage(Handler.java:95)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at android.os.Looper.loop(Looper.java:154)
10-10 22:34:08.953  2793  2808 E GeckoProfile: 	at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)
10-10 22:34:08.953  2793  2793 D GeckoNetworkManager: New network state: UP, CELLULAR, CELL_4G
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 30 ("GeckoBackgroundThread")
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String java.io.File.getAbsolutePath()' on a null object reference
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoApp$10.run(GeckoApp.java:1238)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: 	at android.os.Handler.handleCallback(Handler.java:751)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: 	at android.os.Handler.dispatchMessage(Handler.java:95)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: 	at android.os.Looper.loop(Looper.java:154)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: 	at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler: Main thread (1) stack:
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     java.lang.ref.WeakReference.<init>(WeakReference.java:57)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     org.mozilla.gecko.GeckoActivityMonitor.updateActivity(GeckoActivityMonitor.java:34)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     org.mozilla.gecko.GeckoActivityMonitor.onActivityStarted(GeckoActivityMonitor.java:71)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.Application.dispatchActivityStarted(Application.java:207)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.Activity.onStart(Activity.java:1190)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.support.v4.app.FragmentActivity.onStart(FragmentActivity.java:552)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.support.v7.app.AppCompatActivity.onStart(AppCompatActivity.java:177)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     org.mozilla.gecko.GeckoApp.onStart(GeckoApp.java:1277)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     org.mozilla.gecko.BrowserApp.onStart(BrowserApp.java:1114)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1248)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.Activity.performStart(Activity.java:6679)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2609)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2707)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.ActivityThread.-wrap12(ActivityThread.java)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1460)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.os.Looper.loop(Looper.java:154)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:6077)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
10-10 22:34:08.954  2793  2808 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)

and also reported that once fennec crashed, any later runs would crash also.

I think we should check:
 - does the wpt harness force-stop the browser before trying to start it?
 - does the wpt harness delete and recreate the remote profile before trying to start the browser
 - runtime permission grants

Comment 6

[Geoff Brown [:gbrown]]

A couple of tricks:

mach android-emulator --force-update 

will start Android with a fresh image -- it will blow away any previous state, installed apps, etc.

adb install -g <apk> will install the app with all runtime permissions granted.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5417a7093182
Run the emulator automatically r=gbrown

Comment 8

[Geoff Brown [:gbrown]]

Testing 'mach wpt' today, I had trouble with fennec crashing until I used 'adb install -g <apk>' to install fennec with all runtime permissions granted.

Crashed:

$ ./mach build && ./mach package && ./mach android-emulator --force-update && ./mach install && ./mach -v wpt testing/web-platform/tests/infrastructure/

No crash:

$ ./mach build && ./mach package && ./mach android-emulator --force-update && adb install -g ~/objdirs/x86/dist/fennec-64.0a1.en-US.android-i686.apk && ./mach -v wpt testing/web-platform/tests/infrastructure/


'mach mochitest' and friends avoid the need for 'install -g' by calling grant_runtime_permissions():

https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/testing/mozbase/mozrunner/mozrunner/devices/android_device.py#376

Comment 9

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/5417a7093182

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

grant_runtime_permissions() gets tests running on my MacBook which previously was complaining about host/port issues. 

Patch incoming...

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1496627 - Make sure fennec is granted all the permissions it needs to run wpt

Without this, the user must manually install fennec with expanded permissions with `adb install -g <apk>`.

With this patch, `./mach wpt` on its own handles everything from emulator setup to test running.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/facb6f18068e
Make sure fennec is granted all the permissions it needs to run wpt r=gbrown

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/facb6f18068e

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

With comment 13 merged around (and a workaround for bug 1472814 set up if I'm doing stuff on Debian-based distros), `mach wpt` successfully runs tests.

TODO:
1) Finish running through all of the tests not explicitly prevented from running on fennec. I split them up into 18 chunks and I have chunk 9 running right now.
2) Get a list of failing tests from the test results from (1). Upload these logs somewhere (Google Docs, I guess?).
3) Figure out what needs to be fixed. Some early thoughts:
** Most seem to be passing already, at least with the x86 emulator on a relatively beefy macbook pro. I imagine there might be some timeouts on the slower arm emulators or on less beefy host machines...
** Some tests testing webkit-prefixed things are unexpectedly passing; I assume because fennec emulates support for them to make the Chrome-dominated mobile web nicer for users, while desktop expectations don't?
** I'm seeing some mozlog errors during the test runs.
4) Work with gbrown to get these tests running in CI.
5) While running these tests, the test run for the current chunk ends. I'm able to type new commands into the terminal, and the emulator sits around, not doing anything. If I try to close the terminal, though, it asks me if I want to terminate running processes in the terminal. ("Closing this window will terminate the running process -bash.")
** Should the emulator get shut down when the tests are finished? I don't know if other test suites do anything like that.
** What process is still running? I don't think anything should still be running, and it might explain why I'm seem to need to restart my computer between test runs to make sure things go smoothly.


Anyway, we're done in this bug, I'll work on those other things elsewhere.

Comment 15

[Geoff Brown [:gbrown]]

(In reply to Wes Kocher (:KWierso) from comment #14)
> TODO:
> 1) Finish running through all of the tests not explicitly prevented from
> running on fennec. I split them up into 18 chunks and I have chunk 9 running
> right now.

That's great! Keep in mind that we want to run these in CI in the geckoview TestRunnerActivity rather than fennec. That might affect test results. I'm trying to get TestRunnerActivity (mach wpt --package-name=org.mozilla.geckoview.test) working in bug 1496773 - maybe next week?

> 3) Figure out what needs to be fixed. Some early thoughts:
> ** Most seem to be passing already, at least with the x86 emulator on a
> relatively beefy macbook pro. I imagine there might be some timeouts on the
> slower arm emulators or on less beefy host machines...

The plan is to run these in CI on x86 emulators, hopefully fast. Locally, most developers use the x86 emulator because the arm emulator is so slow. Technically we want to support armv7/aarch64 of course, but I wouldn't worry too much about that.

> ** Should the emulator get shut down when the tests are finished? I don't
> know if other test suites do anything like that.

No, we don't do that normally.

> ** What process is still running? I don't think anything should still be
> running, and it might explain why I'm seem to need to restart my computer
> between test runs to make sure things go smoothly.

Not sure. It seems odd to me.

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Oh, and...

6) I think we should move the verify_android_device and grant_runtime_permissions calls to be later in the wpt process. We should be able to most of the not-actually-running-tests wpt commands without spinning up the emulator.
7) Maybe rerun these tests next week with Bug 1498680 landed to see if that fixes up any failing tests.

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

That still-running process apparently goes away if I just press 'enter' once after the test run finishes, so that's cool.

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

> > ** What process is still running? I don't think anything should still be
> > running, and it might explain why I'm seem to need to restart my computer
> > between test runs to make sure things go smoothly.
> 
> Not sure. It seems odd to me.

Not sure anything's actually still running for real. When the tests all finish, the console goes back to what seems to be the blank prompt for new input. If I try to close the termi al window, I get the question about the process that's still running.
But if I instead hit enter with no other input, the console prints out a blank line before showing me the prompt for input. After that, I can close the terminal window with no prompts.

Unsure what this means.