1043962 - [marionette-js-runner] look into only restarting the host between test files for perf

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[Andrew Halberstadt [:ahal]]

Right now marionette-js-runner will restart the host between each subtest and an additional time between suites. E.g if a single file has 4 subtests in it, the host will get restarted 5 times.

This is ok on b2g desktop, because restarting is really quick. On devices, a restart can take ~30-60 seconds. This makes the test run *really* slow and limits the amount of tests we can run in continuous integration.

One possibility would be to only restart the host in-between test files. This still gives isolation between unrelated tests but greatly reduces the number of times we need to restart. I think the requirement that different tests in the same file play nicely with each other is a fairly reasonable one.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

I think I've seen a number of tests within a file that assume they are starting from a blank slate.  I think it's worth considering also just having an option to blow away the application's cookies/localstorage/IndexedDB between individual tests.  This could speed up b2g-desktop runs too.

Comment 2

[Hubert Figuiere [:hub]]

It happen that when the bug was filed, we would never restart if using the device host runner. See bug 1027232 for details.

But be aware: if you touch this you touch make test-perf at the same time.

Comment 3

[James Lal [:lightsofapollo]]

There are low level optimizations we can do to "restart" in an incomplete but fast way... fabrice worked on this way back but should take way less time then 30s we should figure out how to be fast and realistic (but deterministic tests are more important to me then pure speed).

Comment 4

[BMO Automation]

Firefox OS is not being worked on