670229 - [Tracking bug] Determine which mochitests are 'slow'

Status: NEW

(Testing :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Determine which mochitest-plain tests are slow, so they can be targeted for further investigation.

Comment 1

[Jonathan Griffin (:jgriffin)]

For mozilla-central on 2011-07-07, the timings for all tests (in ms) are here:

http://people.mozilla.org/~jgriffin/mochitest.json

The tests that take greater than 10s are extracted here:

http://people.mozilla.org/~jgriffin/mochitest-more-than-10s.txt

I'll setup a system to collect this data on an ongoing (weekly?) basis.

Comment 2

[Jonathan Griffin (:jgriffin)]

One of the criteria for considering a test as "slow" is if the debug duration of the test is much larger than the opt duration of the test.

Crunching the data above, I come up with these average ratios:

{
  "win32": {
    "debug": 3079053,
    "opt": 849342,
    "ratio": 3.6252216421653469
  },
  "macosx": {
    "debug": 5510793,
    "opt": 95161,
    "ratio": 57.91020481079434
  },
  "macosx64": {
    "debug": 5065500,
    "opt": 145407,
    "ratio": 34.836699746229549
  },
  "linux64": {
    "debug": 4934970,
    "opt": 776520,
    "ratio": 6.3552387575336118
  },
  "linux": {
    "opt": 798284,
    "debug": 5980156,
    "ratio": 7.4912637607668451
  }
}

Each ratio is the number of ms for all the debug tests / ms for all opt tests.  The mac numbers are so high because apparently I didn't capture all the opt mac tests during the period in which this data was saved.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. 

from irc w/jgriffin, this is a tracking bug for work by ATeam to find+fix slow running mochitests (in dependent bugs), so morphing summary and move to Testing:General