1013298 - tp4m has only two samples - median is not a good statistic

Status: RESOLVED FIXED

(Datazilla Graveyard :: Metrics, defect)

Description

[Kyle Lahnakoski [:ekyle]]

dzAlerts uses median value from each test.  tp4m has only two samples per test, which results in the mean == median.  This is a problem because first page load time is significantly different than the second (in most cases it seems).

Either parametrize dzAlerts so we can declare the statistic to run regression analysis on, or split each tp4m test into two sub-tests with one run each.  The latter will benefit us by comparing first load times and subsequent load times.  The former may be of benefit to other tests where median is not appropriate.

Comment 1

[Kyle Lahnakoski [:ekyle]]

Now that we set per-test parameters, I have changed tp4m to use last (last replicate) instead of median.

> "tp4m": {
>     "units": "ms",
>     "select": {
>         "value": {
>             "name": "value",
>             "value": "result.stats.last"
>         }
>     }
> }