851187 - Running large amounts of mochitests makes them run slower and slower

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Martijn Wargers (dead)]

While working on bug 817639, I noticed that mochitests were running increasingly slower on my Macbook Pro Retina, the further along the test run.
At the end of the test run, after 196535 tests, they were running really slow.

I know that the mochitest base page is adding a table-row item for every test that is being run. See also bug 494546.

I did a mochitest run with these 196535 tests with style="display:none" around these table with test results and then the mochitest would run much faster.

Without this style="display:none" hack:
INFO | automation.py | Application ran for: 5:32:27.586478

With this style="display:none" hack:
INFO | automation.py | Application ran for: 1:24:14.365317

So with that style="display:none" hack, this mochitest run was running almost 4 times as fast!

Do we need to have these results displayed in the mochitest base page?
Or could we perhaps suffice by only displaying the failures?
Btw, on b2g mochitest run, I can't even see those results displayed anyway for some reason.

Could bug 815726 be related to this also?

Btw, I think this is also the reason why shutdown takes something like 1 minute or so after the mochitest run.

Comment 1

[Martijn Wargers (dead)]

Attachment: This is what I did in the mochitest base page to make the tests run faster

Comment 2

[(not currently active) Ted Mielczarek]

I thought bug 787916 fixed this?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I believe our mochitest-chrome test suite falls into this trap as well; For the automation, this was fixed at one point in time to display:none (via environment variable I believe) for mochitest-plain.

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Fix went in quite a while ago. Seems like it should have been merged to the b2g branch, but I wonder if it actually was.

Comment 5

[Martijn Wargers (dead)]

Apparently something else is going on, because plain-loop.html isn't even used, afaik. So I guess the fast run was just plain luck or something.

Comment 6

[(not currently active) Ted Mielczarek]

Ed: I'm pretty sure we already fxied this in the general case. There may be something B2G-specific here.

Comment 7

[Martijn Wargers (dead)]

Ok, what I noticed while running dom/tests/mochitest/dom-level2-html/ only, I noticed that every test would take something like 4 seconds.
After I would click somewhere in the b2g window, every test suddenly would take less than 1 second!
Also, after that this message would be gone too: "Unable to restore focus, expect failures and timeouts".
So I guess this has something to do with bug 826147.

Comment 8

[Martijn Wargers (dead)]

So I guess this is some kind of focus issue? Is there some way to really focus the b2g mochitest app?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

could we fix the harness to not require focus (i.e. --ignore-focus).  We could make the harness not retry, currently we do three retries with a 1 second sleep between retry:
http://dxr.mozilla.org/mozilla-central/testing/mochitest/tests/SimpleTest/TestRunner.js#l242

Comment 10

[Jonathan Griffin (:jgriffin)]

(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22)(vacation till april 2nd) from comment #8)
> So I guess this is some kind of focus issue? Is there some way to really
> focus the b2g mochitest app?

Yes, this is a known issue, bug 788866.  I'm not sure what the fix is.  Definitely adding --ignore-focus or some such to the harness would at least let the tests run more quickly.

Comment 11

[Martijn Wargers (dead)]

This was fixed by bug 788866.