820883 - Tests are laggy when there are too many suites and tests in one file

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Julien Wajsberg [:julienw] (PTO -> Mar 16)]

We noticed that having a lot of suites or tests in the same file make the tests very slow.

I tracked the problem to the process.nextTick function. Actually, when executing the test directly in the browser (without the test-agent) I noticed that the tests were executing quite fast but then the browser was left very laggy during one or two minutes. Profiling showed that most of the time was spent in the "message" handler of the "nextTick" function. This led me to think that the problem was similar when launching with the test-agent.

Comment 1

[Julien Wajsberg [:julienw] (PTO -> Mar 16)]

I tried the same thing with a newer mocha and the problem disappeared, however there were some incompatibility in the test-agent that I had to fix.

I also had to modify one method of mocha (look for "Mozilla" in the file) because some methods where missing after a json serialization. There may be a better way but this one worked.

I have problems in some tests which seems to rely on setTimeout and I tried to fix them but I'm not sure all the problems are because of the faster execution.

I'm giving up for now, patch to follow.

Comment 2

[Julien Wajsberg [:julienw] (PTO -> Mar 16)]

Attachment: patch v1

(apply with |git am|)

update mocha, fix test-agent to work with new mocha, fix mocha to work with
test-agent, update update_manager_test so that it kind of works with the faster
execution

Comment 3

[Julien Wajsberg [:julienw] (PTO -> Mar 16)]

James, would love to have your advice on this.

Comment 4

[James Lal [:lightsofapollo]]

I have wanted to update mocha for awhile, given that "nextTick" uses postMessage it is likely somewhat slow in volume. Not sure what they did to fix the problem in the update... I will update test-agent first then move the changes over to gaia.

Comment 5

[James Lal [:lightsofapollo]]

Note there is overhead running with test-agent, tests definitely will be faster if you run them without it. That problem is likely fixable we just need to profile test-agent to figure out where it slows things down.

Comment 6

[Julien Wajsberg [:julienw] (PTO -> Mar 16)]

some measurements from the app_install_manager_test.js in system app.

without this patch :
  ✔ 146 tests complete (14253ms)
  ✔ 254 tests complete (60966ms)
  ✔ 380 tests complete (143352ms)

with this patch:
  ✔ 146 tests complete (1255ms)
  ✔ 254 tests complete (2545ms)
  ✔ 380 tests complete (4211ms)

We can see this is a lot more linear after the mocha update.

There are a lot of nested suites in this test file, and observing the output, it seems that the more a suite is nested, the more the test execution is slow. However, mocha doesn't report any slowness for the tests so it must be the time between the tests that is longer.


Also, I noticed that a console.log was forgotten in the patch :)

Comment 7

[James Lal [:lightsofapollo]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7051

Pointer to Github pull-request

Comment 8

[James Lal [:lightsofapollo]]

Comment on attachment 693085 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7051

Mostly the same from your patch but updates test-agent directly (in its own repo). I published v0.8 globally and it is fully compatible with mocha 1.7x.

There may be some issues using the newer mocha methods but existing tests pass (or fail) the same way they did prior to the patch.

I also added a simple pending test for fm radio.

Comment 9

[Etienne Segonzac (:etienne)]

https://github.com/mozilla-b2g/gaia/commit/c527532bfcadbc5ca89efd1340f3630bb1419889