1500509 - Move Android marionette tests to Android 7.0 x86_64

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P3)

Description

[Geoff Brown [:gbrown]]

Currently, Android marionette tests run on our Android 4.3 API16+ test platform. They run in 10 test chunks.

We can run the same tests on the new Android 7.0 x86 test platform, so that
 - tests run on a more modern, more relevant OS;
 - tests run much faster.


Android 7.0 x86 is available on try and mozilla-central now, but is not yet available on integration branches; we'll need more task capacity before proceeding with any changes here.

Comment 1

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0e500574ac2ebce0cbd702957fe0331140a6f4e5 

demonstrates how this would work. Note that all tests can be run in one chunk in just 14 minutes.

There are a few failing tests -- investigation required.

Comment 2

[Chris Peterson [:cpeterson]]

Geoff, do you plan to eventually move all the Fennec tests from the Android 4.3 API16+ test platform to Android 7.0 x86 (on Packet)?

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Chris Peterson [:cpeterson] from comment #2)
> Geoff, do you plan to eventually move all the Fennec tests from the Android
> 4.3 API16+ test platform to Android 7.0 x86 (on Packet)?

No, I don't, at least not in the near future. 

Once the existing Android 7.0 x86 tests are Tier 1, I hope we can consider simply turning off some Android 4.3 API16+ tests which provide similar coverage (like geckoview-junit, jsreftests and crashtests). The remaining tests we'd need to consider on a case-by-case basis: We know there's an issue with running xpcshell on Android 7.0; we expect robocop to have a lot of intermittents on any new test platform; etc. I don't foresee having the time to resolve those issues.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

:gbrown, is there specific work needed to move marionette?  from the earlier comment there were some failures.

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #0)
> Android 7.0 x86 is available on try and mozilla-central now, but is not yet
> available on integration branches; we'll need more task capacity before
> proceeding with any changes here.

This is no longer a concern.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #4)
> :gbrown, is there specific work needed to move marionette?  from the earlier
> comment there were some failures.

Yes, there are some failures that need to be addressed.

Also, we should run these (and all the x86 tests) against x86-64 opt/debug builds, so we should get that sorted out first - bug 1498298. (Changing the build may affect which tests fail, so fix the 64 bit builds, then look into greening up.)

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Geoff, does something else still block us, or can we proceed with this bug? Would be great to see us moving away from the old emulator.

Comment 8

[Geoff Brown [:gbrown]]

There are some test failures in the new environment. I reported one with an obvious fix, but I don't have much time to look into the others just now. Otherwise, I think we are ready to move.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6fc83f52ed252f5174290da9233fa6fa2c8b2508

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I filed two additional bugs for those remaining two cases. I think that we should be able to switch over soon.

Comment 10

[Geoff Brown [:gbrown]]

I still see frequent "Port 127.0.0.1:2828 is unavailable.":

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=ef1cde90bb0924b70bd1082700b4b210ce6f1114

Most of those go away if we run in more chunks:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=96d9af0ef1c380dcce2241176cf1a4616f7672ee

I'd like to figure out what is causing that.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Good question. At least the Android logcat isn't that particular helpful. I wasn't able to find anything related to this connection problem. Could it be some instability with adb and port forwarding? Maybe that just drops at some random interval?

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So the crashes (bug 1527254) will be fixed soon. Geoff, could you push another try build once it has been done? Lets see what's remaining.

Comment 13

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=844455a30bdf2242a0ffd177663303f901eb55fa

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Note that I commented on the remaining blocking bugs for making this transition real. All of them look like issues with adb or the emulator but not the Marionette tests. When I run the tests locally in a x86-7.0 emulator it works all fine, and no failures are present.

Comment 15

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3546c1d4a25cbfc8585238ef60cc4e268f9fdbeb

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With the recent try push only the screen orientation failures are blocking us from moving over to the x86_86 emulator. I will try to get this done today.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1500509 - Move marionette tests to Android 7.0 x86_64

Depends on D21358

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 9047040 [details]
Bug 1500509 - Move marionette tests to Android 7.0 x86_64

Sorry, moz-phab uploaded this temporarily applied patch. I don't want to steal Geoff's work.

Comment 19

[Geoff Brown [:gbrown]]

Attachment: Bug 1500509 - Move Android marionette tests to 7.0 x86_64; r=whimboo

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4348e60c81ed
Move Android marionette tests to 7.0 x86_64; r=whimboo

Comment 21

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/4348e60c81ed

Comment 22

[Geoff Brown [:gbrown]]

Checking a few pushes before/after this change, I estimate Android 4.3 Mn was taking about 300 minutes of aws time per push (all chunks on opt and debug combined) and Android 7.0 is taking about 20 minutes of packet.net time per push.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I noticed that too, which is an amazing improvement for resource usage and costs. It's great to see that this is done now.