Created attachment 8470441 [details] [diff] [review] counttests.diff On some platforms, with some mochitests, you see different amount of subtests running, occasionally. Ideally, these tests should always run the same amount of subtests. I made a poc patch, but it doesn't really make much sense. It's rather easy to test by doing is(SimpleTest._tests.length, 5, "wrong!") for example. Btw, also the order in which the subtests are running should be deterministic. I guess there is no easy way to get these kinds of checks of the currently running mochitests on tinderbox?
Do you have an example which illustrates this problem?
I don't have a concrete example of this problem occurring, but I got suspicious while fixing some of the media tests in bug 1032878. For instance in content/media/webspeech/recognition/test/test_call_start_from_end_handler.html I had do make some changes to prevent that test to spew out subresults after SimpleTest.finish(). That led me to think, what guarantee is there that these kinds of tests with event handlers (onplaying on video for example) are giving the correct amount of subresults as the test author would think they would generate. Some of the intended event handlers might have not fired at all and the mochitest would still pass (with less subresults). I suspect there might be quite a few tests that show different amount of result after every run. I guess it would easiest, if you could compare tbpl runs somehow.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
You need to log in before you can comment on or make changes to this bug.