1051229 - Need a way to ensure that correct amount of subtests are run every time

Status: REOPENED

(Testing :: Mochitest, defect)

Description

[Martijn Wargers (dead)]

Attachment: counttests.diff

On some platforms, with some mochitests, you see different amount of subtests running, occasionally.
Ideally, these tests should always run the same amount of subtests.

I made a poc patch, but it doesn't really make much sense.
It's rather easy to test by doing is(SimpleTest._tests.length, 5, "wrong!") for example.

Btw, also the order in which the subtests are running should be deterministic.

I guess there is no easy way to get these kinds of checks of the currently running mochitests on tinderbox?

Comment 1

[Jonathan Griffin (:jgriffin)]

Do you have an example which illustrates this problem?

Comment 2

[Martijn Wargers (dead)]

I don't have a concrete example of this problem occurring, but I got suspicious while fixing some of the media tests in bug 1032878.
For instance  in content/media/webspeech/recognition/test/test_call_start_from_end_handler.html I had do make some changes to prevent that test to spew out subresults after SimpleTest.finish().
That led me to think, what guarantee is there that these kinds of tests with event handlers (onplaying on video for example) are giving the correct amount of subresults as the test author would think they would generate. Some of the intended event handlers might have not fired at all and the mochitest would still pass (with less subresults).

I suspect there might be quite a few tests that show different amount of result after every run.
I guess it would easiest, if you could compare tbpl runs somehow.