406906 - should parallelize talos tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dietrich Ayala (:dietrich)]

We currently run all of our unit tests on a single tinderbox, which leads to very long cycles. The test suites are not interdependent, do not need to run serially.

Shorter cycles will yield test results faster, increasing developer productivity.

Comment 1

[Dietrich Ayala (:dietrich)]

Changing summary, as this is applicable to perf tests as well. Eg: The Talos boxes have super long cycles. We could probably have Talos Tp run on it's own box, and group the rest. Alice, are the Talos tests interdependent or can they each run stand-alone?

Comment 2

[Rob Campbell [:rc] (:robcee)]

Regarding the unittest boxes, it makes some sense to split up Tunit, reftest and mochitest and various browser/chrome frameworks. How we end up doing that, having one machine responsible for the build, and then 2 or more machines downstream that can pick up, copy that build or if 3 machines perform their own build step (in parallel) prior to running the test is something we'd have to figure out.

Assuming the buildtime is constant, we're only talking about a savings of about 5-10 minutes on all platforms by parallelizing. Our slowest box, qm-winxp01 is a VM and could (should?) be replaced by physical hardware. Possibly the same for qm-centos5-01.

For talos, we have a slightly different problem. These machines aren't creating their own builds so we don't have to worry about that. What is a concern is the amount of infrastructure we'll need to copy in order to parallelize. If we split Ts, Txul, TDHTML etc. off of Tp, that means we'll need to multiply our infrastructure by 2. We're already seeing requests for chromeless implementations of talos running alongside the existing set. Does this imply a factor of 4 for hardware? Just curious what we're looking at here.

Comment 3

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.

Comment 4

[[e-mail] (:bhearsum)]

Dietrich, is this still a pain-point?

Comment 5

[Chris Cooper [:coop] (he/him)]

Alice also has bug 473821 for speeding up talos.

Comment 6

[Dietrich Ayala (:dietrich)]

un-duping, per catlee, because bug 452861 is about *unit tests* and this is about perf tests.

Comment 7

[Chris AtLee [:catlee]]

Looks like we need to split up buildbot jobs to have different values passed to --activeTests.

Also, need to figure out how we would report all the different results.  Maybe per-branch Talos trees on tinderbox?  Like Firefox3.5-Talos, analogous to the Firefox3.5-Unittest.

Comment 8

[alice nodelman [:alice] [:anode]]

This ended up getting done as a side effect of buildbot cleanup work - we run tp4/dromaeo as a separate job on separate machines.  We'll split out more test sets as they get added to talos.

Comment 9

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.