Investigate running packet.net-based Android emulator unit tests against x86-64 build

RESOLVED FIXED in Firefox 67

Status

enhancement
P3
normal
RESOLVED FIXED
7 months ago
6 days ago

People

(Reporter: gbrown, Assigned: gbrown, NeedInfo)

Tracking

Version 3
mozilla67
x86_64
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 wontfix, firefox65 wontfix, firefox66 wontfix, firefox67 fixed)

Details

(Whiteboard: [geckoview:p2])

Attachments

(2 attachments, 1 obsolete attachment)

Assignee

Description

7 months ago
If we switch our packet.net geckoview tests from the "Android 4.2 x86 opt" build to the "Android 5.0 x86-64 opt" build, how do the test results compare?
Assignee

Updated

7 months ago
Assignee: nobody → gbrown
Assignee

Comment 2

7 months ago
In the above try push, tests run against the "Android 5.0 x86-64 opt" build.

Performance of all test suites seems very similar to normal packet.net x86.

mochitest and geckoview-junit test results are the same as on normal packet.net x86.

*** crashtest and jsreftests have persistent crashes not seen with x86 builds. ***


:jchen -- Just making sure you are aware of these results; follow-up as you desire!
Flags: needinfo?(nchen)
Assignee

Updated

7 months ago
No longer depends on: 1489264, 1490040, 1490962
See Also: → 1494388
Assignee

Updated

7 months ago
Blocks: 1425322
Assignee

Updated

7 months ago
Assignee: gbrown → nobody
Priority: P1 → P3
OS: Unspecified → Android
Hardware: Unspecified → x86_64
Flags: needinfo?(nchen)
Assignee

Comment 3

6 months ago
Updated, with proper e10s support (wpt excluded). Includes x86-64 opt and debug now:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3852a308e3b85f4fdc5c7e10d128a950a062e341
Assignee

Updated

5 months ago
Blocks: 1500509
Assignee

Updated

5 months ago
Depends on: 1512427
Assignee

Comment 4

5 months ago
I'll land some task configuration changes now that will make this easier to switch over in the future (but won't make any actual changes to today's tasks).

Comment 6

5 months ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/135c1ea8854e
Prepare to run packet.net geckoview tests against x86_64 opt/debug builds; r=me,a=test-only
Assignee

Comment 7

5 months ago
This should be all that's needed to switch to x86_64 and introduce x86_64/debug tests. Before landing, we need to green up tests (investigate crashes).

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=769eb3ffa0efccbd4be900784101827b54c5864f

Comment 8

5 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/135c1ea8854e
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
Assignee

Updated

5 months ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee

Updated

4 months ago
Depends on: 1518849
Assignee

Comment 10

4 months ago

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

Assignee: nobody → gbrown

(In reply to Geoff Brown [:gbrown] from comment #10)

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

If I'm reading Treeherder correctly, are the results reversed? It looks like the crashtests (C) are green on opt but orange on debug and the jsreftests (J1-J8) are green on both opt and debug.

I filed bug 1524493 for the debug crashtest failure.

Assignee

Comment 12

4 months ago

(In reply to Chris Peterson [:cpeterson] from comment #11)

(In reply to Geoff Brown [:gbrown] from comment #10)

Comment 9 is good news indeed: crashtests run green on opt and debug; jsreftests run green on opt.

If I'm reading Treeherder correctly, are the results reversed? It looks like the crashtests (C) are green on opt but orange on debug and the jsreftests (J1-J8) are green on both opt and debug.

Yes, that's correct. Thanks!

Assignee

Updated

4 months ago
Depends on: 1524693
Assignee

Comment 13

4 months ago

Let's do this:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=ef8fa87e327af92d0de25d902e15420aaf7434ef

  • stop running x86/opt mochitest-cl, mochitest-gpu, crashtest, jsreftest, geckoview-junit
  • start running x86_64/opt mochitest-cl, mochitest-gpu, crashtest, jsreftest, geckoview-junit
  • start running x86_64/debug mochitest-cl, mochitest-gpu, jsreftest
  • no /debug crashtests until at least bug 1524493 is resolved
  • no /debug geckoview-junit, and geckoview-junit runs tier 3, until at least bug 1521195 is resolved
  • web-platform tests remain on x86 for now; perhaps move them to x86_64 when they can run on geckoview
  • treeherder labels for x86_64 will be tidied up in bug 1524693
Attachment #9034505 - Attachment is obsolete: true
Attachment #9040860 - Flags: review?(jmaher)
Attachment #9040860 - Flags: feedback?(snorp)
Attachment #9040860 - Flags: feedback?(cpeterson)
Comment on attachment 9040860 [details] [diff] [review]
switch most tests to x86_64/opt and add x86_64/debug tests too

LGTM! Looking at your Treeherder results, I see:

x86_64 opt: crashtests, jsreftests, mochitest-clipboard, and mochitest-gpu are all green.

x86_64 debug: jsreftests, mochitest-clipboard, and mochitest-gpu are all green. Debug crashtests are disabled due to bug 1524493.
Attachment #9040860 - Flags: feedback?(cpeterson) → feedback+
Duplicate of this bug: 1501579
Comment on attachment 9040860 [details] [diff] [review]
switch most tests to x86_64/opt and add x86_64/debug tests too

Review of attachment 9040860 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/test-sets.yml
@@ +394,2 @@
>      - web-platform-tests
>      - web-platform-tests-reftests

So the only tests that will still be running on Android x86 will be the web-platform-tests?
Assignee

Comment 17

4 months ago

(In reply to Chris Peterson [:cpeterso

So the only tests that will still be running on Android x86 will be the
web-platform-tests?

Correct!

Comment on attachment 9040860 [details] [diff] [review]
switch most tests to x86_64/opt and add x86_64/debug tests too

Review of attachment 9040860 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/reftest.yml
@@ +69,3 @@
>      chunks:
>          by-test-platform:
>              android-em-4.3-arm7-api-16/debug: 100

do we get any real value from jsreftests on android 4.3 arm?  this seems like a very expensive cost and I would question the use of it.  but unrelated to this bug!

::: taskcluster/ci/test/test-sets.yml
@@ +389,3 @@
>      # - mochitest-webgl1-core
>      # - reftest
>      # - test-verify

why are these commented out?  waiting on greening up?  I was under the impression these would be running on opt/debug.
Attachment #9040860 - Flags: review?(jmaher) → review+

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #18)

do we get any real value from jsreftests on android 4.3 arm? this seems
like a very expensive cost and I would question the use of it. but
unrelated to this bug!

@ Sean, how important are the jsreftests compared to the jit-tests?

On our Google Pixel 2 test devices, we currently run the jit-tests (for both ARMv7 and ARM64 Gecko builds) but not jsreftests.

On our Android ARMv7 emulator, we currently run jsreftests but not jit-tests.

Assignee

Comment 20

4 months ago

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #18)

do we get any real value from jsreftests on android 4.3 arm? this seems
like a very expensive cost and I would question the use of it. but
unrelated to this bug!

We're scheduled to consider turning off Android 4.3 jsreftests once this bug is resolved and we have seen a few days of x86_64 results; will pursue that in bug 1473368.

::: taskcluster/ci/test/test-sets.yml
@@ +389,3 @@

 # - mochitest-webgl1-core
 # - reftest
 # - test-verify

why are these commented out? waiting on greening up? I was under the
impression these would be running on opt/debug.

Yes, waiting on greening up. Most of that work is tracked in bug 1460411.

Attachment #9040860 - Flags: feedback?(snorp) → feedback+

Comment 21

4 months ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/908a14f72ae3
Switch most Android 7.0 x86 tests to x86_64 builds; add Android 7.0 x86_64/debug tests; r=jmaher
Status: REOPENED → RESOLVED
Last Resolved: 5 months ago3 months ago
Resolution: --- → FIXED
Target Milestone: mozilla66 → mozilla67
Assignee

Updated

3 months ago
See Also: → 1526521
See Also: → 1552051
You need to log in before you can comment on or make changes to this bug.