1166576 - test_collapse.html is perma-fail but the mochitest summary shows 0 fails

Status: NEW

(Testing :: Mochitest, defect)

Description

[(no longer active)]

This test is currently run as part of ICS Emulator B2G Debug runs in mochitest-11.  It fails but as you can see in <http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-67d305caa2a8/try-emulator-debug/try_ubuntu64_vm-b2g-emulator-debug_test-mochitest-debug-11-bm53-tests1-linux64-build314.txt.gz> for example, it doesn't cause the tree to go orange.

Comment 1

[Jonathan Griffin (:jgriffin)]

This is probably a mochitest bug; the test summary is showing no failures:

17:26:25     INFO -  351 INFO TEST-START | Shutdown
17:26:25     INFO -  352 INFO Passed:  28046
17:26:25     INFO -  353 INFO Failed:  0
17:26:25     INFO -  354 INFO Todo:    2571

This causes the mozharness script to terminate with a pass.

Comment 2

[Jonathan Griffin (:jgriffin)]

This test uses its own subharness, testharness.js, and it's likely the problem lies in some interaction with that and mochitest.

Comment 3

[(no longer active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> This test uses its own subharness, testharness.js, and it's likely the
> problem lies in some interaction with that and mochitest.

Ms2ger may know about that.

Comment 4

[Martijn Wargers (dead)]

Here is the list with expected failures in for the test_collapse.html file:
http://mxr.mozilla.org/mozilla-central/source/dom/imptests/failures/editing/selecttest/test_collapse.html.json

Comment 5

[(no longer active)]

(In reply to Martijn Wargers [:mwargers] (QA) from comment #4)
> Here is the list with expected failures in for the test_collapse.html file:
> http://mxr.mozilla.org/mozilla-central/source/dom/imptests/failures/editing/
> selecttest/test_collapse.html.json

This is the test timing out, that failure isn't listed there...

Comment 6

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Tried to reproduce locally:

10 INFO Passed:  42
11 INFO Failed:  4
12 INFO Todo:    10