1361863 - SETA: Consider tracking jobs that are optimized out consistently

Status: RESOLVED INVALID

(Tree Management Graveyard :: Treeherder: SETA, enhancement)

Description

[Robert Wood [:rwood]]

An idea mentioned by :davehunt in the Automationeers meeting: Maybe we could have SETA also track test jobs that have been constantly optimized out / are always deemed low value.

Possibly such data could be used to help analyze test jobs themselves and determine if the job itself could be improved or if possibly it is of no value to run anymore. Investigate if we could have SETA track such jobs and/or provide an API to return such data.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

the biggest concern here is that SETA typically optimizes out chunks or certain jobs on certain platforms.  So the tests find issues, but instead of running it on all platforms we run it on one platform every push.  Sadly the problem with SETA is that we optimize out chunks 1,2,3 and run 4,5- but then our tests move between chunks- this is why we keep a 90 day rolling window which helps to minimize some of the problems.

For tests like jsreftest, crashtest, gtest, cpp, jittest, mn, wr, wd, clipboard, gpu, webgl- there are fewer chunks and we could consider this approach :)  When we do 'run by manifest' then this would be an awesome idea!

Comment 2

[Armen [:armenzg]]

We could consider this on a re-write. It is non-trivial.