1517351 - Permafailing Tier 2 Android 8.0 TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/wasm/spec/address.wast.js | #1 module successfully instantiated: PASS. with wasm-testharness.js:31:15 Error: Assertion failed: got false, expected true

Status: RESOLVED FIXED

(Core :: JavaScript: WebAssembly, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&searchStr=android%2C8.0%2Cpixel2%2Copt%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fopt-jittest-10%2C%28jit10%29&revision=5510614c90e62a810e3c0c92934afba301b09315

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219636117&repo=mozilla-inbound&lineNumber=12310

19:48:10     INFO -  #334 module successfully instantiated: PASS.
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 Error: #335 A wast test that runs without any special assertion.: FAIL.
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 RuntimeError: index out of bounds
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 @/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js line 141 > WebAssembly.Module:wasm-function[0]:0x84
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 @/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js line 141 > WebAssembly.Module:wasm-function[1]:0x54
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 call@/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js:213:46
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 @/data/local/tests/jit-tests/jit-tests/tests/wasm/spec/address.wast.js:657:11
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 run@/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js:240:18
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15 @/data/local/tests/jit-tests/jit-tests/tests/wasm/spec/address.wast.js:657:1
19:48:10     INFO -  /data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15
19:48:10     INFO -  Stack:
19:48:10     INFO -    test@/data/local/tests/jit-tests/jit-tests/lib/wasm-testharness.js:31:15
19:48:10     INFO -    uniqueTest@/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js:152:5
19:48:10     INFO -    run@/data/local/tests/jit-tests/jit-tests/lib/../tests/wasm/spec/harness/sync_index.js:244:5
19:48:10     INFO -    @/data/local/tests/jit-tests/jit-tests/tests/wasm/spec/address.wast.js:657:1Exit code: 3
19:48:10     INFO -  FAIL - wasm/spec/address.wast.js
19:48:10  WARNING -  TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/wasm/spec/address.wast.js | #1 module successfully instantiated: PASS. (code 3, args "") [0.3 s]
19:48:10     INFO -  {"action": "test_start", "jitflags": "", "pid": 431, "source": "jittests", "test": "wasm/spec/address.wast.js", "thread": "main", "time": 1546458490.197213}
19:48:10     INFO -  {"action": "test_end", "extra": {"jitflags": ""}, "jitflags": "", "message": "#1 module successfully instantiated: PASS.", "pid": 431, "source": "jittests", "status": "FAIL", "test": "wasm/spec/address.wast.js", "thread": "main", "time": 1546458490.500932}
19:48:10     INFO -  INFO exit-status     : 3


:Ms2ger can you please take a look? Thank you.

Comment 1

[Intermittent Failures Robot]

22 failures in 312 pushes (0.071 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 12
* autoland: 8
* mozilla-central: 2

Platform breakdown:
* android-hw-p2-8-0-android-aarch64: 1
* android-hw-p2-8-0-arm7-api-16: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1517351&startday=2019-01-02&endday=2019-01-02&tree=all

Comment 2

[Andreea Pavel [:apavel]]

There are 46 total failures in the last 7 days, all on android-hw-p2-8-0-arm7-api-16 opt.

Comment 3

[Intermittent Failures Robot]

24 failures in 425 pushes (0.056 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 6
* autoland: 12
* mozilla-central: 6

Platform breakdown:
* android-hw-p2-8-0-arm7-api-16: 22
* android-hw-p2-8-0-android-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1517351&startday=2019-01-03&endday=2019-01-03&tree=all

Comment 4

[Intermittent Failures Robot]

67 failures in 1668 pushes (0.04 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 27
* autoland: 28
* mozilla-central: 12

Platform breakdown:
* android-hw-p2-8-0-arm7-api-16: 64
* android-hw-p2-8-0-android-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1517351&startday=2018-12-31&endday=2019-01-06&tree=all

Comment 5

[Bob Clary [:bc] (inactive)]

dluca: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tier=1%2C2%2C3&searchStr=android-hw&revision=752c683e631d73518b21da4b0924ac80ce1f6d5f&selectedJob=220576455 is not this bug.

It has

exception output: /data/local/tests/jit-tests/jit-tests/tests/auto-regress/bug710192.js:10:1 Error: Assertion failed: got (void 0), expected "1234"
TEST-UNEXPECTED-FAIL | tests/jit-test/jit-test/tests/basic/bug1355573.js | Segmentation fault (code 139, args "") [5.8 s]

which is not this bug.

Comment 6

[Bob Clary [:bc] (inactive)]

ebalazs_: See comment 5.

Comment 7

[Eliza Balazs [:ebalazs_]]

bc: Thank you, We'll file a bug for the new failure.

Comment 8

[Bob Clary [:bc] (inactive)]

Attachment: bug-1517351-disable-jit10-tests-arm.patch

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3e26b297e547ecc40e2dd32dbb127c73ac594643

Comment 9

[Andreea Pavel [:apavel]]

Joel can we land this?

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/72c3b5930c7d
Disable jit-test/tests/wasm/spec/{address.wast,call_indirect.wast}.js on arm-32, r=jmaher

Comment 11

[Lars T Hansen [:lth]]

Thanks, I did see that. I've been monitoring this bug for a couple of days and I'm fairly worried, but I have very few spare cycles at the moment :( Will try to dig into it in the morning.

Comment 12

[Bob Clary [:bc] (inactive)]

Sorry, I didn't get to it right away. Tree closure and a meeting DOSed me.

Comment 13

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/72c3b5930c7d

Comment 14

[Intermittent Failures Robot]

19 failures in 542 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 11
• autoland: 2
• mozilla-inbound: 6

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1517351&startday=2019-01-08&endday=2019-01-08&tree=all

Comment 15

[Lars T Hansen [:lth]]

Failure is caused by the test labeled "// address.wast:535" even when run in isolation. That is this test case:

(module
  (type (;0;) (func))
  (type (;1;) (func (param i32) (result f32)))
  (import "$3" "32_good1" (func (;0;) (type 1)))
  (func (;1;) (type 0)
    block  ;; label = @1
      i32.const 65525
      call 0
      i32.reinterpret_f32
      f32.const 0x0p+0 (;=0;)
      i32.reinterpret_f32
      i32.eq
      i32.eqz
      br_if 0 (;@1;)
      return
    end
    unreachable)
  (export "run" (func 1)))

and the function "32_good1" comes from here:

(module
  (type (;0;) (func (param i32) (result f32)))
  (type (;1;) (func (param i32)))
  (func (;0;) (type 0) (param i32) (result f32)
    local.get 0
    f32.load)
  (func (;1;) (type 0) (param i32) (result f32)
    local.get 0
    f32.load align=1)
  (func (;2;) (type 0) (param i32) (result f32)
    local.get 0
    f32.load offset=1 align=1)
  (func (;3;) (type 0) (param i32) (result f32)
    local.get 0
    f32.load offset=2 align=2)
  (func (;4;) (type 0) (param i32) (result f32)
    local.get 0
    f32.load offset=8)
  (func (;5;) (type 1) (param i32)
    local.get 0
    f32.load offset=4294967295
    drop)
  (memory (;0;) 1)
  (export "32_good1" (func 0))
  (export "32_good2" (func 1))
  (export "32_good3" (func 2))
  (export "32_good4" (func 3))
  (export "32_good5" (func 4))
  (export "32_bad" (func 5))
  (data (;0;) (i32.const 0) "\00\00\00\00\00\00\a0\7f\01\00\d0\7f"))

This looks like an f32 load from an unaligned address without any alignment specifier; a known dark corner of our ARM implementation...

OOB is a strange error to see here since the access is clearly in-bounds, so more likely the load causes an exception that is interpreted by us as OOB. But we'll see.

Comment 16

[Lars T Hansen [:lth]]

Basically, HandleTrap() treats SIGSEGV and SIGBUS the same, so a SIGBUS alignment trap will be treated as a SIGSEGV out-of-bounds trap, which is why we see OOB here.

It would appear from this test file that this is only an issue for floating point accesses, since the integer tests earlier in the file did not fail; this again suggests the Android kernel does some kind of filtering of alignment errors, since the ARMv8 only has a single control word bit for alignment checking; if set, everything is checked; if unset, only some instructions (like atomics) are checked. The filtering would amount to fixing up integer accesses but letting floating point accesses flow through to a SIGBUS. Joy.

The chipset on the Pixel 2 is a Snapdragon 835 which has a Kryo 280 cpu which is a modified ARMv8 Cortex-A73, supposedly.

Comment 17

[Lars T Hansen [:lth]]

I take back some of the analysis in comment 16, since this is ARMv7 code and it has a more complicated trap model, and FP accesses can trap independently of int accesses. "It's complicated." ARMv8 manual E2.4.

Comment 18

[Lars T Hansen [:lth]]

In principle, something like this in HandleTrap:

  if (isSigbus) {
    // Faulting memory access.
#ifdef __arm__
    uint32_t instr = *(uint32_t*)pc;
    if ((instr & 0x0F300E00) == 0x0D100A00) { // VLDR
      bool isDouble = (instr & 0x00000100) != 0;
      bool add = (instr & 0x00800000) != 0;
      bool dBit = (instr >> 22) & 1;
      bool offs = (instr & 0xFF) << 2;
      bool rn = (instr >> 16) & 0xF;
      MOZ_ASSERT(rn != 15);     // Or we must align it
      uint8_t* p = ReadGPR(context, rn) + (add ? offs : -offs);
      // Do we need to worry about whether p is in bounds?  As in, is SEGV
      // handled before BUS or not?
      if (isDouble) {
        bool vd = ((instr >> 12) & 0xF) | (dBit << 4);
        double val;
        memcpy(&val, p, sizeof(val));
        WriteFPR64(context, vd, val);
      } else {
        bool vd = ((instr >> 11) & (0xF << 1)) | dBit;
        float val;
        memcpy(&val, p, sizeof(val));
        WriteFPR32(context, vd, val);
      }
    }
#endif // __arm__
  }

However, the WriteFPR() functions may not be easily implementable, in
https://android.googlesource.com/platform/bionic/+/c124baa/libc/arch-arm/include/machine/ucontext.h we read that the FP state is not really accessible and is anyway kernel-configuration dependent. So it's uncertain whether we can do a good job this way.

An alternative is to set up an exception return to code that performs the emulation back in userland, where the registers are right. That might not be so bad.

Comment 19

[Intermittent Failures Robot]

36 failures in 2627 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #31 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 14
• mozilla-inbound: 9
• mozilla-central: 13

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1517351&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 20

[Andreea Pavel [:apavel]]

There are no failures in the last 7 days, the ones from the above comment are from before the disable patch got landed.

Comment 21

[Lars T Hansen [:lth]]

This (and several others) will be closed the minute we're sure the patch for bug 1283121 is not backed out :)