649286 - Pagesets in Talos environment slow down add-on tests unnecessarily

Status: RESOLVED INVALID

(Testing :: Talos, defect)

Description

[Wladimir Palant]

In bug 647398 comment 9 Jorge states that available hardware is currently limiting the number of tests that can run. What strikes me about this is that testing an add-on takes 5-6 minutes, a single machine could test 100 add-ons in 10 hours (in reality four machines would need to run 10 hours of course because of four operating systems). The remaining 8-10 minutes are spent setting up Talos environment, especially pagesets (more than 2 minutes alone for unpacking pagesets.zip, also a significant proportion of time in "chmod files" and "remove old working dirs" steps that together take more than 5 minutes). Yet this data isn't used in add-on performance measurements - by not adding it to the environment you should be able to halve the time spent for testing an add-on.

Comment 1

[alice nodelman [:alice] [:anode]]

We can disable the pageset download, it was initially set up during an early iteration of testing where we were also running tp.

The hardware blocking issue are in bug 599169 and have to do with setting up a separate addon testing pool - which will also allow us to turn on additional tests.

Comment 2

[Wladimir Palant]

I don't really see how this is an answer to my suggestion. But if you prefer throwing more hardware at the problem as the only solution then so be it.

Comment 3

[alice nodelman [:alice] [:anode]]

It's a solution because we had an agreement with release engineering to do testing for the top 100 addons on their system, should we want to do more testing we need to get on our own testing pool.

Comment 4

[alice nodelman [:alice] [:anode]]

It's a solution because we had an agreement with release engineering to do testing for the top 100 addons on their system, should we want to do more testing we need to get on our own testing pool.