690219 - Need to track intermittent talos oranges

Status: RESOLVED WORKSFORME

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[cmtalbert]

There are likely two phases to this request.  The first is a course phase to detect catastrophic talos failure.  The second is to watch for actual talos oranges like any other orange on a normal test and to track it in the ES database so we can determine if the occurrences of the talos orange tests are getting better or worse.

This is specifically being requested due to difficulties on android, so if you want to turn on the feature just for android talos then that's fine.

I think Joel summed it up best in an email - this describes what we'd like for a phase 1:
"Another tool that might be nice is using the orange factor toolchain to analyze tests and report daily if we have an abnormal amount of failures for a given test.  Say the last 5 days tp4m failed 25% of the time, but today it failed 90% of the time- that should be a orange flag!"

This would be useful, because there is very little to distinguish the talos failures from one another.  So, doing this course analysis for phase 1 would be helpful in the short term.

Phase 2:
Actually analyzing the talos log files and determining the rate of occurrence for specific intermittent failures.  This is going to be difficult because the talos logs aren't formatted particularly well for this type of analysis, and it is hard to tell if failure x is an instance of bug a, b, or c.  This may require better output from talos, so it might need to be spun into a bug/project of its own.

Comment 1

[Phil Ringnalda (:philor)]

/me drifts off to sleep nightly with visions of better-formatted talos failures dancing in his head