1315749 - Try to find a way to run at least a sanity browser test for Android x86

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P2)

Description

[Geoff Brown [:gbrown]]

We currently run xpcshell on Android x86 builds, but no browser tests. As demonstrated recently by bug 1315512, that means it is possible to break Firefox on Android x86 without anyone noticing for a while.

We have had trouble running mochitests, reftests, etc reliably on the x86 emulator in automation - the emulator becomes unresponsive intermittently without any apparent warning or error in logs. We put a lot of time into trying to debug that condition, but never found a cause or solution and ultimately gave up, noting low usage of the x86 builds.

I also recently tried to use a newer x86 emulator and/or Android image in bug 1309892, but without success (emulator will not complete boot on aws).

We could try:
 - a simpler mochitest - just a sanity test, or something like that
 - using autophone

Comment 1

[Nick Alexander :nalexander [he/him]]

Thanks for this valuable context, gbrown.  I was wondering why this was the case, and now I know!  It's such a pity, since the sweet spot for developers is either a modern ARM device or the x86 emulator.

Comment 2

[Bob Clary [:bc] (inactive)]

FWIW: Autophone does support x86 builds.

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Geoff Brown [:gbrown] from comment #0)
> We currently run xpcshell on Android x86 builds, but no browser tests. As
> demonstrated recently by bug 1315512, that means it is possible to break
> Firefox on Android x86 without anyone noticing for a while.
> 
> We have had trouble running mochitests, reftests, etc reliably on the x86
> emulator in automation - the emulator becomes unresponsive intermittently
> without any apparent warning or error in logs. We put a lot of time into
> trying to debug that condition, but never found a cause or solution and
> ultimately gave up, noting low usage of the x86 builds.

This actually seems to be well understood.  The x86 emulator uses KVM, and all major cloud providers don't support nested KVM.  See https://www.ravellosystems.com/blog/android-emulator-on-amazon-ec2-and-google-cloud/ and many other similar discussions.

Sadly, that appears to mean we can either pay to use some non-standard snowflake, or we don't run an x86 emulator in AWS.

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Nick Alexander :nalexander (leave until January 2017) from comment #3)
> The x86 emulator uses KVM,

Does it always? I know there is a qemu -enable-kvm option (which we don't use on aws), but does the x86 emulator need/use kvm if that option is not specified? I always assumed not.

Comment 5

[Geoff Brown [:gbrown]]

I thought mochitest-chrome might be a simple, short job that would do the job here. It runs fairly well:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=617d86b8c7d196b88d0b480564fa27daa368c97e

But I keep seeing infrequent crashes, never in the same test twice; I don't think I can proceed until those are resolved.

:snorp - Can you have a look at these crashes?

https://queue.taskcluster.net/v1/task/dRffk2rDRIqxLpJXjyDoTA/runs/0/artifacts/public/logs/live_backing.log

https://queue.taskcluster.net/v1/task/WNNg6oKLSne1gQXbIm6t6w/runs/0/artifacts/public/logs/live_backing.log

https://queue.taskcluster.net/v1/task/QeZP4Hb8T3msc_wR3UXDQA/runs/0/artifacts/public/logs/live_backing.log

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

Geoff I think this is some sort of timeout issue? I see:

[task 2016-11-18T00:37:27.228709Z] 00:37:27     INFO -  mozdevice ERROR | Timeout exceeded for shell call 'adb shell dumpsys window windows; echo $?'

A bit later:

[task 2016-11-18T00:37:32.625281Z] 00:37:32     INFO -  org.mozilla.fennec still alive after SIGABRT: waiting...

And then the process dies from the SIGABRT.

Comment 7

[Geoff Brown [:gbrown]]

Yes, I see now. I'll investigate.

Comment 8

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d13432af39a719f89edae42be69cc9aae9b897f

Comment 9

[Geoff Brown [:gbrown]]

Attachment: add x86 mochitest-chrome

See try push in comment 8.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8813244 [details] [diff] [review]
add x86 mochitest-chrome

Review of attachment 8813244 [details] [diff] [review]:
-----------------------------------------------------------------

nice work here.

Comment 11

[Nick Alexander :nalexander [he/him]]

gbrown: thanks for jumping on this!  You might be interested in https://bugzilla.mozilla.org/show_bug.cgi?id=1318820 locally, since it'll speed up any local testing you're doing.

jmaher: thanks for prompt reviews!

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7a74c4841f0a
Run mochitest-chrome on Android x86; r=jmaher

Comment 13

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/7a74c4841f0a