700815 - Showing average test times is pretty useless: should show average of maximum test times instead

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: GoFaster, defect)

Description

[William Lachance (:wlach)]

Right now the dashboard has this view for seeing what the average time for a test is:

http://brasstacks.mozilla.com/gofaster/#/executiontime/test

In actual fact, this is kinda useless though, as our main interest is in test times that are holding us back. It doesn't matter if our average is 30 minutes (e.g.) per test if there's a single test that regularly takes 90 minutes.

I propose that we revise this metric to be an average of maximum test times (that is, take the maximum test time on each platform for each run, and then divide that by the number of runs in the day).

Comment 1

[Armen [:armenzg]]

Perhaps pointing the name of the worst offender also helps (see bug 700815).