1098508 - Run Android jit-tests

Status: REOPENED

(Testing :: General, defect)

Description

[Dan Minor [:dminor]]

These tests are currently orange on Cedar but the js team is planning to apply resources to get these fixed so we can schedule these elsewhere.

I'll file blockers on this bug for the current failures.

Comment 1

[Dan Minor [:dminor]]

Needinfo'ing terrence to keep this on his radar.

Comment 2

[Dan Minor [:dminor]]

Is there a bug on file for the madness seen here [1]?

[1] https://tbpl.mozilla.org/?tree=Try&rev=e6f9f7154b36&showall=1

Comment 3

[Dan Minor [:dminor]]

Ah yes, good ol bug 882179.

Comment 4

[Phil Ringnalda (:philor)]

No, the bug for that would be an unfiled "jittest on Android needs to handle timeouts internally rather than allowing 'mozdevice.devicemanager.DMError: Automation Error: Timeout in command execcwd' to be raised, since that infinitely retries." The bug is not that infinite retries are a possibility, it's that this suite on this platform is broken by not short-circuiting what would be infinite retries in every suite on android, if not for the way that every other suite takes care of timing out without winding up with a devicemanager error that retries.

Comment 5

[Dan Minor [:dminor]]

(In reply to Phil Ringnalda (:philor) from comment #4)
> No, the bug for that would be an unfiled "jittest on Android needs to handle
> timeouts internally rather than allowing 'mozdevice.devicemanager.DMError:
> Automation Error: Timeout in command execcwd' to be raised, since that
> infinitely retries." The bug is not that infinite retries are a possibility,
> it's that this suite on this platform is broken by not short-circuiting what
> would be infinite retries in every suite on android, if not for the way that
> every other suite takes care of timing out without winding up with a
> devicemanager error that retries.

Filed bug 1098529 to fix this.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Bug 1098529 has been fixed for a long time. Where do we stand on getting JIT tests running on Android 4.3?

Comment 7

[Dan Minor [:dminor]]

Assuming this work is still relevant, I think gbrown would be the person to ask as I don't work on mobile stuff anymore. Chances are no one has mentioned getting the jit-tests running on 4.3 to him :)

Comment 8

[Geoff Brown [:gbrown]]

I know nearly nothing about jit-tests. I should rectify that!

I'll schedule a trial run on 4.3 next week to see what needs to be done.

Comment 9

[Geoff Brown [:gbrown]]

jit-tests masquerade as cppunit in https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b6ec02c344a. It looks like this is a long-running test and will require at least 6 chunks. Most tests pass.

Comment 10

[Geoff Brown [:gbrown]]

jit-tests run as 10 chunks, masquerading as mochitests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=233b836bcb42

Comment 11

[Geoff Brown [:gbrown]]

It looks like Android 4.3 Opt jit-tests would need to run as 10 chunks, with each job running in about 40 minutes, on c3.xlarge aws instances. On Android 4.3 Debug, jit-tests would need about 20 chunks.

Most tests pass, but there are at least 10 tests failing consistently. I do not see open bugs for these failures.

Comment 12

[Geoff Brown [:gbrown]]

:terrence -- Are jit-tests still desired on Android? How urgent, how important are they? We need someone to investigate and resolve test failures on Android before moving forward. Once tests run green on Android, there may also be a capacity issue, since it looks like these would need to run in many chunks (see details in my comments above).

Comment 13

[Terrence Cole [:terrence]]

(In reply to Geoff Brown [:gbrown] from comment #12)
> :terrence -- Are jit-tests still desired on Android? How urgent, how
> important are they? We need someone to investigate and resolve test failures
> on Android before moving forward. Once tests run green on Android, there may
> also be a capacity issue, since it looks like these would need to run in
> many chunks (see details in my comments above).

It's sad that it takes over a compute-day to run these tests, but the existence of 10(!) previously unknown failures rather proves the point that these are *desperately* needed, no?

As to performance, we need to measure how much is harness and exec overhead -- maybe adding a fork server mode could cut down on the number of chunks.

Comment 14

[Terrence Cole [:terrence]]

The failures here appear to be mostly in SIMD (see the tbpl link in comment 10). Are these known? Are there fixes in the pipeline?

Comment 15

[Benjamin Bouvier [:bbouvier] (inactive)]

I'll take care of the SIMD tests, these were unknown and need fixing, thanks for letting me know. However, most of the failing tests are *not* related to SIMD, see list below :)

All the failing tests are the following:
- jit-tests/jit-tests/tests/SIMD/shuffle.js (for nbp or me)
- jit-tests/jit-tests/tests/SIMD/swizzle.js

- jit-tests/jit-tests/tests/asm.js/testAtomics.js (that'd be for lth)

- jit-test/jit-test/tests/basic/spread-call-maxarg.js (i'd say jit-es6 features are for jandem, or efaust, without any certainty)
- jit-test/jit-test/tests/basic/spread-call-near-maxarg.js

- jit-test/tests/ctypes/conversion-finalizer.js (don't know who takes care of ctypes these days? sfink?)
- jit-test/jit-test/tests/ctypes/conversion-native-function.js

- jit-test/jit-test/tests/basic/bug1195452.js
- jit-test/jit-test/tests/basic/bug832203.js

- jit-test/jit-test/tests/gc/bug-1143706.js (terrence, jonco, sfink)
- jit-tests/tests/gc/incremental-abort.js

- jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType-unknown.js (Typed objects are probably for bhackett now)
- jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType.js 

- jit-test/jit-test/tests/ion/inlining/isFiniteInline.js (till or h4writer, probably)
- jit-test/jit-test/tests/ion/inlining/isNaNInline.js

Comment 16

[Nicolas B. Pierron [:nbp]]

(In reply to Terrence Cole [:terrence] from comment #14)
> The failures here appear to be mostly in SIMD (see the tbpl link in comment
> 10). Are these known? Are there fixes in the pipeline?

I am not aware of these failure, at least not while running the test suite under the ARM simulator.

Geoff, can you paste/attach the /proc/cpuinfo of these phones?

Comment 17

[Nicolas B. Pierron [:nbp]]

(In reply to Benjamin Bouvier [:bbouvier] from comment #15)
> -
> jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType-
> unknown.js (Typed objects are probably for bhackett now)
> - jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType.js 

These test are doing a timeout when I run them under the jit-test, but I recall seeing them finishing without.

Comment 18

[Geoff Brown [:gbrown]]

The tests shown on treeherder are running in the Android arm emulator running Android 4.3. The emulator is running on an aws c3.xlarge instance. I'll fetch the /proc/cpuinfo from the emulator for you...

Comment 19

[Geoff Brown [:gbrown]]

00:58:17     INFO - Running timeout 30 /builds/slave/test/build/android-sdk18/platform-tools/adb -s emulator-5554 shell cat /proc/cpuinfo
00:58:17     INFO - Processor	: ARMv7 Processor rev 0 (v7l)
00:58:17     INFO - BogoMIPS	: 602.93
00:58:17     INFO - Features	: swp half thumb fastmult vfp edsp neon vfpv3 
00:58:17     INFO - CPU implementer	: 0x41
00:58:17     INFO - CPU architecture: 7
00:58:17     INFO - CPU variant	: 0x0
00:58:17     INFO - CPU part	: 0xc08
00:58:17     INFO - CPU revision	: 0
00:58:17     INFO - 
00:58:17     INFO - Hardware	: Goldfish
00:58:17     INFO - Revision	: 0000
00:58:17     INFO - Serial		: 0000000000000000
00:58:17     INFO - Processor	: ARMv7 Processor rev 0 (v7l)
00:58:17     INFO - BogoMIPS	: 602.93
00:58:17     INFO - Features	: swp half thumb fastmult vfp edsp neon vfpv3 
00:58:17     INFO - CPU implementer	: 0x41
00:58:17     INFO - CPU architecture: 7
00:58:17     INFO - CPU variant	: 0x0
00:58:17     INFO - CPU part	: 0xc08
00:58:17     INFO - CPU revision	: 0
00:58:17     INFO - 
00:58:17     INFO - Hardware	: Goldfish
00:58:17     INFO - Revision	: 0000
00:58:17     INFO - Serial		: 0000000000000000

Comment 20

[Nicolas B. Pierron [:nbp]]

NI-self: We should check with the simulator, by setting the ARMHWCAP environment variable to the list of cpu features reported in comment 19.

Comment 21

[Nicolas B. Pierron [:nbp]]

(In reply to Geoff Brown [:gbrown] from comment #19)
> /builds/slave/test/build/android-sdk18/platform-tools/adb -s emulator-5554
> shell cat /proc/cpuinfo
> 00:58:17     INFO - Processor	: ARMv7 Processor rev 0 (v7l)
> 00:58:17     INFO - BogoMIPS	: 602.93
> 00:58:17     INFO - Features	: swp half thumb fastmult vfp edsp neon vfpv3 

While running the test suite with an optimized build running the ARM simulator, I was able to see 2 unexpected JS assertion:

 - asm.js/testCaching.js
 - asm.js/testBullet.js

These 2 failures disappear when I add "vfpv4" to the hardware feature list.

Comment 22

[Nicolas B. Pierron [:nbp]]

Attachment: Propagate ARM_HWCAP command line options to the nested shell.

This patch fixes the issue seen in comment 21, which is related to the fact
that the cache was not working due to different hardware capabilities.

Unfortunately, this is unlikely to be fixing any of the test failure
reported in comment 15, and I am unable to reproduce any of these in the ARM
simulator (even with the CPU features listed in comment 19).

Comment 23

[Nicolas B. Pierron [:nbp]]

Lars, by chance, were you able to reproduce any of the comment 15 failures on real hardware?

Comment 24

[Benjamin Bouvier [:bbouvier] (inactive)]

Comment on attachment 8677480 [details] [diff] [review]
Propagate ARM_HWCAP command line options to the nested shell.

Review of attachment 8677480 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks. Maybe we should propagate all the other ARM flags as well. Or maybe it doesn't matter that much. Maybe.

Comment 25

[Lars T Hansen [:lth]]

- jit-tests/jit-tests/tests/asm.js/testAtomics.js

Very likely fixed a couple of weeks back - code generation bug, illegal instruction format.

- jit-test/jit-test/tests/basic/spread-call-maxarg.js
- jit-test/jit-test/tests/basic/spread-call-near-maxarg.js

These two are https://bugzilla.mozilla.org/show_bug.cgi?id=1204605, qv for more detail.

- jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType-unknown.js
- jit-test/jit-test/tests/ion/inlining/TypedObject-TypeDescrIsArrayType.js 

These two were fixed yesterday (by moving them to manual-tests).

Comment 26

[Benjamin Bouvier [:bbouvier] (inactive)]

Out of curiosity, why was this closed? This still sounds like a valuable target to reach, even if no actions have been taken recently towards that goal.

Comment 27

[Geoff Brown [:gbrown]]

I would like to see it kept open. I think "Android 10+" may have made this sound obsolete. Also, we're not ready to schedule the tests, so let's make the bug title more generic. Let's move it to another component too...not sure we need releng involvement at this point.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2fc3d46a16e8
Add support for scheduling Android jit tests, but don't run them yet; r=me,a=test-only

Comment 29

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/2fc3d46a16e8

Comment 30

[Geoff Brown [:gbrown]]

:bc is currently looking into running jittest on Android hardware, including some changes in bug 1440714. He is also seeing test failures, which are being looked at in bug 1454918.

Assuming that work goes ahead, we can probably give up on the idea of jittest on the Android emulator, and dup this bug to one of bc's.