928356 - Ideas for saving time in test runs

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[Zac C (:zac)]

Our test run might get 30 minutes longer soon.

Throw your ideas in here.. what have you got?

Removing any redundant framework steps?

Merging any very similar/duplicate tests?

Shifting code to only run where it is needed?

Comment 1

[Bob Silverberg [:bsilverberg]]

What about removing tests from the suite that runs on device, if those tests are also running on desktop? How often (if ever) have we caught a regression that only appears on device but not desktop?

As a follow on to the above, getting tests to run on the emulator, which will open up more possibilities of moving some tests off device and onto emulator.

Comment 2

[Florin Strugariu [:Bebe]]

Can we run tests on multiple devices at the same time?
Could we split the test run in smaller parts run a basic "smoke" run and after that run some other tests... That might help us see the results more quickly

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] from comment #2)
> Can we run tests on multiple devices at the same time?
> Could we split the test run in smaller parts run a basic "smoke" run and
> after that run some other tests... That might help us see the results more
> quickly

We already run tests on multiple devices, but I'm assuming you mean run a single testrun across multiple devices in parallel? We currently specify the host/port of the Marionette server on the command, and assume there is a single device attached for ADB commands. Also, the test runner currently runs tests sequentially and not in parallel, so I believe this would be a considerable amount of work.

Comment 4

[Zac C (:zac)]

Yes I was thinking it would be a load easier to just have hamachi.m-c.ui.1 and hamachi.m-c.ui.2 and split the manifests in a manner that makes them 50%/50% time wise. I am not mad on extra jobs as always but the hamachi.m-c.ui job is the only one for which it is needed because of its high importance.

The --timeout patch that bitgeeky is working should save a little bit of time but only when there are actually timeouts so we can expect only single digit minute savings, at best.

I think we might be able to merge a few more clock tests a bit more; that creates more of a saving.


Something that Dave proposed a few months ago was to switch the `sleep(0.5)` in the waits to occur at the end of the while loop but it's hard to quantify how much this will actually save compared to the risk of introducing intermittents, I am much more worried about the latter.

Comment 5

[Florin Strugariu [:Bebe]]

We made some difs and using the new marionette Wait's will save a lot of time...

What do you guys think if we migrate to the new waits?

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Florin Strugariu [:Bebe] from comment #5)
> We made some difs and using the new marionette Wait's will save a lot of
> time...
> 
> What do you guys think if we migrate to the new waits?

It's in progress. See bug 959217.

Comment 7

[Zac C (:zac)]

Three things:
1. split run into two
2. Dave's work on Bug 959217 
3. work on Bug 955626 will save >5 sec per startup.