737886 - Post alerts when significant changes are detected in test results

Status: RESOLVED WONTFIX

(Testing Graveyard :: Peptest, defect)

Description

[Mark Côté [:mcote]]

Graphing the results of the buildbot peptest runs is only half of the battle; we need alerts posted (to dev.tree-management and perhaps elsewhere) when there are significant deviations in the results of a peptest test. The main difficulty will be in detecting real deviations.

Comment 1

[Mark Côté [:mcote]]

I looked at things like exponential smoothing algorithms to eliminate noise. Being a bit out of my depth, I then talked to the analytics team.

Their first question was about the 50-ms cut-off. I had to admit that I didn't know if this was made up or based on something from published literature. I assume there is some justification about it being the perceptual boundary for smooth UI interactions.

However a bigger issue for them is that we don't collect any data on event-loop durations for values under 50 ms. This, to them, is incomplete data, and it makes it very hard for them to put into context those periods that we *do* record (Gilbert used the example of comparing murder rates in New York versus his hometown in Ireland without knowing the population size of either). A specific concern was for values that hover around 50 ms; half the time we might be recording values of, say, 51 or 52, and the other half we wouldn't be recording them at all because they'd fall just below 50.

They requested (very gently that, if at all possible, we provide them with full data on event-loop durations for a test run or two in order to characterize the data. I think maybe a try run or two, with the threshold turned down to 0 ms, should suffice.

Comment 2

[cmtalbert]

I think at this point we can wontfix this and reopen if we ever re-invigorate peptest again?

Comment 3

[Mark Côté [:mcote]]

Agreed.