1339843 - consider adjusting SETA to detemine high value jobs as jobs which don't necessarily catch every regression

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: Treeherder: SETA, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

there are many cases where we have a single test job that only catches a single regression.  While this served its purpose, we could greatly reduce the number of jobs we mark high value if we ignore single instances.  Many jobs are responsible for catching 2,3,4,5+ regressions- those are super high value.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this reduces the jobs from 160->82, so cut in half.  That gives us 83% of historical failures detected, most likely this will cause a few more misses, but we guarantee that every 5th push we will run all the jobs.

Not sure if this is worth pursuing, but it would be low cost, maybe there are other ways to skin this cat.