679894 - Failure in test_load.html occurs after SimpleTest.finish(), and thus is hidden in TBPL

Status: RESOLVED WORKSFORME

(Core :: Audio/Video, defect)

Description

[Jonathan Griffin (:jgriffin)]

While digging through test failures using the OrangeFactor database, I discovered a failure in test_load.html that isn't being reported in TBPL.  This happens because TBPL relies on the mochitest summary, and this failure is missing from the summary, presumably because it happens after SimpleTest.finish() is called.

One instance of this is here:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1312206659.1312207600.4850.gz&fulltext=1#err0

You can see that the test ends, and then causes a failure which seems to get lost because it happens after test end:

68747 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - corrent number of events received
68748 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - events received in order - "source_error" should equal "source_error"
68749 INFO TEST-END | /tests/content/media/test/test_load.html | finished in 898ms
68750 INFO TEST-START | /tests/content/media/test/test_load_candidates.html
68751 INFO TEST-PASS | /tests/content/media/test/test_load_candidates.html | Test seek.webm 5 - corrent number of events received
68752 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_load_candidates.html | Test seek.webm 5 - events received in order - got "error", expected "loadstart"

Note that the error is actually produced by test_load.html, not test_load_candidates.html as shown in the log.

Compare with a recent successful log:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1313607839.1313608638.30498.gz&fulltext=1

69522 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - corrent number of events received
69523 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - events received in order - source_error should equal source_error
69524 INFO TEST-PASS | /tests/content/media/test/test_load.html | Should only get error events here
69525 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - corrent number of events received
69526 INFO TEST-PASS | /tests/content/media/test/test_load.html | Test seek.webm 5 - events received in order - source_error should equal source_error
69527 INFO TEST-END | /tests/content/media/test/test_load.html | finished in 552ms
69528 INFO TEST-START | /tests/content/media/test/test_load_candidates.html
69529 INFO TEST-PASS | /tests/content/media/test/test_load_candidates.html | Should have received 2 error events before loaded - 2 should equal 2
69530 INFO TEST-PASS | /tests/content/media/test/test_load_candidates.html | Should have received 2 error events before loaded - 2 should equal 2
69531 INFO TEST-PASS | /tests/content/media/test/test_load_candidates.html | Should have received 2 error events before loaded - 2 should equal 2
69532 INFO TEST-PASS | /tests/content/media/test/test_load_candidates.html | Should have received 2 error events before loaded - 2 should equal 2
69533 INFO TEST-END | /tests/content/media/test/test_load_candidates.html | finished in 280ms

Comment 1

[Martijn Wargers (dead)]

Would this still be a problem?
On Windows, there is now a perma-orange, filed as bug 1024535, but the error messages there seem to genuinely coming from test_load_candidates.html itself.

Here, it seems that mochitest is reporting test_load_candidates.html to cause the failure, while it should be test_load.html, which is really bad and I hope that doesn't happen anymore.

In general, I hope to get the "results after SimpleTest.finish should cause failures" fixed in bug 1032878.

Comment 2

[Martijn Wargers (dead)]

I think this bug is WORKSFORME, but I don't dare to close it myself. Johnathan, do you agree this can be closed?

Comment 3

[Jonathan Griffin (:jgriffin)]

Sounds reasonable.