710788 - Recalibrate talos TDHTML on fedora for improved accuracy.

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Stephen Lewchuk]

The current regime of only doing 5 runs is insufficient to detect changes with confidence.  As the result of a simulation effort the following recalibration is proposed to allow 1% regressions to be detected 95% of the time after a single run of Talos.

            component sample_size
1        layers2.html           8
2        layers4.html           6
3        layers5.html          14
4        mozilla.html           7
5        slidein.html          15
6      colorfade.html          18
7       diagball.html          15
8    fadespacing.html          12
9  replaceimages.html          21
10          zoom.html          21

The remaining 8 components of TDHTML, due to the nature of their distributions were unable to reach the 95% detection rate without at least 40 and some time many more runs.

For these components until better detection is available a repetition rate of 21 (the same as the highest of those that reached 95%) is recommended.

These numbers will change if different thresholds (1% change and 95% chance of detection) are chosen.

This implementation will require changes to the structure of the pageloader and how talos executes a test suite, to allow per component and per operating system repetition values.  These changes have the potential to impact the distributions of the test results and this effect should be studied before performing the recalibration.

Comment 1

[Stephen Lewchuk]

I have revised the calibration numbers, after realizing the confidence intervals for some of those values were quite large (+/- 5).  Here are the updated values, with the upper and lower 95% confidence interval on the numbers.  Those with Inf did not reach 95% with less than 30 samples.

            test_name lower mean upper
1      colorfade.html    24   25    26
2       diagball.html    18   19    19
3    fadespacing.html    12   12    12
4     imageslide.html   Inf  Inf   Inf
5        layers1.html   Inf  Inf   Inf
6        layers2.html     8    8     8
7        layers4.html     6    6     6
8        layers5.html    11   11    11
9        layers6.html   Inf  Inf   Inf
10         meter.html   Inf  Inf   Inf
11    movingtext.html   Inf  Inf   Inf
12       mozilla.html     7    7     7
13 replaceimages.html    26   27    27
14     scrolling.html   Inf  Inf   Inf
15       slidein.html    14   15    15
16  slidingballs.html   Inf  Inf   Inf
17     textslide.html   Inf  Inf   Inf
18          zoom.html    19   19    20

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we have turned off tdhtml on all branches.