Move Android marionette tests to Android 7.0 x86_64

RESOLVED FIXED in Firefox 67

Status

defect
P3
normal
RESOLVED FIXED
10 months ago
3 months ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

(Depends on 3 bugs)

Version 3
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox67 fixed)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Currently, Android marionette tests run on our Android 4.3 API16+ test platform. They run in 10 test chunks.

We can run the same tests on the new Android 7.0 x86 test platform, so that
 - tests run on a more modern, more relevant OS;
 - tests run much faster.


Android 7.0 x86 is available on try and mozilla-central now, but is not yet available on integration branches; we'll need more task capacity before proceeding with any changes here.
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0e500574ac2ebce0cbd702957fe0331140a6f4e5 

demonstrates how this would work. Note that all tests can be run in one chunk in just 14 minutes.

There are a few failing tests -- investigation required.
Geoff, do you plan to eventually move all the Fennec tests from the Android 4.3 API16+ test platform to Android 7.0 x86 (on Packet)?
Flags: needinfo?(gbrown)
(In reply to Chris Peterson [:cpeterson] from comment #2)
> Geoff, do you plan to eventually move all the Fennec tests from the Android
> 4.3 API16+ test platform to Android 7.0 x86 (on Packet)?

No, I don't, at least not in the near future. 

Once the existing Android 7.0 x86 tests are Tier 1, I hope we can consider simply turning off some Android 4.3 API16+ tests which provide similar coverage (like geckoview-junit, jsreftests and crashtests). The remaining tests we'd need to consider on a case-by-case basis: We know there's an issue with running xpcshell on Android 7.0; we expect robocop to have a lot of intermittents on any new test platform; etc. I don't foresee having the time to resolve those issues.
Flags: needinfo?(gbrown)
Blocks: 1473368
:gbrown, is there specific work needed to move marionette?  from the earlier comment there were some failures.
Flags: needinfo?(gbrown)
(In reply to Geoff Brown [:gbrown] from comment #0)
> Android 7.0 x86 is available on try and mozilla-central now, but is not yet
> available on integration branches; we'll need more task capacity before
> proceeding with any changes here.

This is no longer a concern.
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #4)
> :gbrown, is there specific work needed to move marionette?  from the earlier
> comment there were some failures.

Yes, there are some failures that need to be addressed.

Also, we should run these (and all the x86 tests) against x86-64 opt/debug builds, so we should get that sorted out first - bug 1498298. (Changing the build may affect which tests fail, so fix the 64 bit builds, then look into greening up.)
Depends on: 1498298
No longer depends on: 1452652
Flags: needinfo?(gbrown)
Depends on: 1520226
No longer depends on: 1520226
Priority: -- → P3

Geoff, does something else still block us, or can we proceed with this bug? Would be great to see us moving away from the old emulator.

Flags: needinfo?(gbrown)
Depends on: 1526084
Summary: Move Android marionette tests to Android 7.0 x86 → Move Android marionette tests to Android 7.0 x86_64

There are some test failures in the new environment. I reported one with an obvious fix, but I don't have much time to look into the others just now. Otherwise, I think we are ready to move.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6fc83f52ed252f5174290da9233fa6fa2c8b2508

Flags: needinfo?(gbrown)
Depends on: 1526913
Depends on: 1526914

I filed two additional bugs for those remaining two cases. I think that we should be able to switch over soon.

Depends on: 1527254

I still see frequent "Port 127.0.0.1:2828 is unavailable.":

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=ef1cde90bb0924b70bd1082700b4b210ce6f1114

Most of those go away if we run in more chunks:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=96d9af0ef1c380dcce2241176cf1a4616f7672ee

I'd like to figure out what is causing that.

Good question. At least the Android logcat isn't that particular helpful. I wasn't able to find anything related to this connection problem. Could it be some instability with adb and port forwarding? Maybe that just drops at some random interval?

So the crashes (bug 1527254) will be fixed soon. Geoff, could you push another try build once it has been done? Lets see what's remaining.

Flags: needinfo?(gbrown)
Depends on: 1527959
Depends on: 1527963

Note that I commented on the remaining blocking bugs for making this transition real. All of them look like issues with adb or the emulator but not the Marionette tests. When I run the tests locally in a x86-7.0 emulator it works all fine, and no failures are present.

With the recent try push only the screen orientation failures are blocking us from moving over to the x86_86 emulator. I will try to get this done today.

Attachment #9047040 - Attachment is obsolete: true

Comment on attachment 9047040 [details]
Bug 1500509 - Move marionette tests to Android 7.0 x86_64

Sorry, moz-phab uploaded this temporarily applied patch. I don't want to steal Geoff's work.

Assignee: nobody → gbrown
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4348e60c81ed
Move Android marionette tests to 7.0 x86_64; r=whimboo
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67

Checking a few pushes before/after this change, I estimate Android 4.3 Mn was taking about 300 minutes of aws time per push (all chunks on opt and debug combined) and Android 7.0 is taking about 20 minutes of packet.net time per push.

I noticed that too, which is an amazing improvement for resource usage and costs. It's great to see that this is done now.

Depends on: 1533084
No longer blocks: 1487124
No longer depends on: 1526914
You need to log in before you can comment on or make changes to this bug.