932492 - Put [DEFAULT] blocks into each functional test folder's manifest file

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[James Lal [:lightsofapollo]]

Right now we are looking at 25 minute test times for gaia-ui-tests. Our dedicated travis queue just kicked in so we now have the capacity to split up our tests in such away that will bring down the overall test times per commit.

My though is to split up https://github.com/lightsofapollo/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/manifest.ini this manifest into 3-5 smaller manifests broken up roughly so each takes a 3-10 minutes.

Comment 1

[Zac C (:zac)]

I would rather not; this doesn't play nicely with our on-device testing and reporting.

I think it will make reporting and debugging a pain on Travis too.

I know some ways to save at least 3 minutes on the current test run anyway.

Comment 2

[James Lal [:lightsofapollo]]

Our build times are only going to get higher- We can write some script to _generate_ the split manifest perhaps but we need to get our overall build times down to a manageable number (this also effects automation).

That aside this should also boost everyone's productivity- 10 minutes is better but my goal is 2 minutes.

Comment 3

[Zac C (:zac)]

It sounds like you mean there will be 3-5 different console outputs? This is the bit I don't like.

Faster test runs I do like :)

Comment 4

[James Lal [:lightsofapollo]]

For the moment this does mean separate outputs. I think the mode with the best UI would be to run one app manifest per vm but that might be too slow. Eventually we need some nicer then travis UI (treeherder) but that is also further out in the future.

Comment 5

[Zac C (:zac)]

If we put [DEFAULT] blocks into each test app/section's manifest then you get this effect by specifying multiple manifests on the command line.

Then we can retain the overall manifest too.

Comment 6

[James Lal [:lightsofapollo]]

Nice! that sound like what we need

Comment 7

[Zac C (:zac)]

*Task*

For each app's manifest in the `tests/functional/` folder, make sure it has a [DEFAULT] block and correct settings for all of the default tags so that it can operate independently of the main manifest file.

Some already have working [DEFAULT] blocks so won't need any work.

Comment 8

[[:AndreiH]]

Attachment: PR https://github.com/mozilla-b2g/gaia/pull/13221

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

While having the defaults in all of the manifests is a good idea, I also like the idea of having a smart way to split the tests. We do this with other suites but by maintaining separate manifest files. One idea would be to have a command line argument such as --split=1:5 which would split the tests discovered into five equal parts and only run part 1.

Comment 10

[Zac C (:zac)]

https://github.com/mozilla-b2g/gaia/commit/bf8ec40b13f61b914de238659df069cae2324f44