1430180 - run tests by component instead of test suite

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I have outlined some experiments done to run tests by component instead of by test suite:
https://elvis314.wordpress.com/2017/12/21/running-tests-by-bugzilla-component-instead-of-test-suite/

while this is fairly easy to hack up, it is a bunch of hacks and there are some big problems to solve.

Here are some of the problems:
1) getting a list of bugzilla components
2) mapping bugzilla components to tests (my patch maps to manifests- not all test types use manifests)
3) I used a trick to dynamically create test jobs in taskcluster- this doesn't support all the options such as: machine type, e10s/!e10s, branch support, chunking support, tier support, etc.  What I think we need to support is machine type (i.e. xlarge, gpu,etc.) and chunking.  
4) the UI in treeherder will look quite messy when it comes to chunking- although I predict a small number of chunks
5) this relies upon mozharness to run each possible test suite all the time- sadly this doesn't include all test types, specifically talos and web-platform-tests; we will need bigger changes to mozharness or separate jobs.
6) the treeherder symbols/codes are hacky- we need a better process for mapping these
7) the data in tree needs to be validated with each component owner
8) I imagine android, coverage, stylo-disabled, etc. will present interesting challenges
9) until we get off buildbot, we cannot do reftests on windows 10, nor talos
10) currently we would run all the jsreftests on every commit- ignoring the logic added to taskcluster to run when certain files change


With all this said, there are probably other things I have overlooked- with some time on this we can start getting a solution that is less hacky and ready for production.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: WIP patch to run by component