Running large amounts of mochitests makes them run slower and slower

RESOLVED FIXED

Status

Testing
Mochitest
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: Martijn Wargers (dead), Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildfaster:?][capacity])

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
While working on bug 817639, I noticed that mochitests were running increasingly slower on my Macbook Pro Retina, the further along the test run.
At the end of the test run, after 196535 tests, they were running really slow.

I know that the mochitest base page is adding a table-row item for every test that is being run. See also bug 494546.

I did a mochitest run with these 196535 tests with style="display:none" around these table with test results and then the mochitest would run much faster.

Without this style="display:none" hack:
INFO | automation.py | Application ran for: 5:32:27.586478

With this style="display:none" hack:
INFO | automation.py | Application ran for: 1:24:14.365317

So with that style="display:none" hack, this mochitest run was running almost 4 times as fast!

Do we need to have these results displayed in the mochitest base page?
Or could we perhaps suffice by only displaying the failures?
Btw, on b2g mochitest run, I can't even see those results displayed anyway for some reason.

Could bug 815726 be related to this also?

Btw, I think this is also the reason why shutdown takes something like 1 minute or so after the mochitest run.
(Reporter)

Comment 1

5 years ago
Created attachment 724999 [details] [diff] [review]
This is what I did in the mochitest base page to make the tests run faster
I thought bug 787916 fixed this?
I believe our mochitest-chrome test suite falls into this trap as well; For the automation, this was fixed at one point in time to display:none (via environment variable I believe) for mochitest-plain.
Fix went in quite a while ago. Seems like it should have been merged to the b2g branch, but I wonder if it actually was.
(Reporter)

Comment 5

5 years ago
Apparently something else is going on, because plain-loop.html isn't even used, afaik. So I guess the fast run was just plain luck or something.

Updated

5 years ago
Whiteboard: [buildfaster:?][capacity]
Ed: I'm pretty sure we already fxied this in the general case. There may be something B2G-specific here.
(Reporter)

Comment 7

5 years ago
Ok, what I noticed while running dom/tests/mochitest/dom-level2-html/ only, I noticed that every test would take something like 4 seconds.
After I would click somewhere in the b2g window, every test suddenly would take less than 1 second!
Also, after that this message would be gone too: "Unable to restore focus, expect failures and timeouts".
So I guess this has something to do with bug 826147.
Depends on: 826147
(Reporter)

Comment 8

5 years ago
So I guess this is some kind of focus issue? Is there some way to really focus the b2g mochitest app?
could we fix the harness to not require focus (i.e. --ignore-focus).  We could make the harness not retry, currently we do three retries with a 1 second sleep between retry:
http://dxr.mozilla.org/mozilla-central/testing/mochitest/tests/SimpleTest/TestRunner.js#l242
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22)(vacation till april 2nd) from comment #8)
> So I guess this is some kind of focus issue? Is there some way to really
> focus the b2g mochitest app?

Yes, this is a known issue, bug 788866.  I'm not sure what the fix is.  Definitely adding --ignore-focus or some such to the harness would at least let the tests run more quickly.
(Reporter)

Updated

5 years ago
Depends on: 788866
(Reporter)

Comment 11

5 years ago
This was fixed by bug 788866.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.