Open Bug 1430180 Opened 4 years ago Updated 4 years ago
run tests by component instead of test suite
I have outlined some experiments done to run tests by component instead of by test suite: https://elvis314.wordpress.com/2017/12/21/running-tests-by-bugzilla-component-instead-of-test-suite/ while this is fairly easy to hack up, it is a bunch of hacks and there are some big problems to solve. Here are some of the problems: 1) getting a list of bugzilla components 2) mapping bugzilla components to tests (my patch maps to manifests- not all test types use manifests) 3) I used a trick to dynamically create test jobs in taskcluster- this doesn't support all the options such as: machine type, e10s/!e10s, branch support, chunking support, tier support, etc. What I think we need to support is machine type (i.e. xlarge, gpu,etc.) and chunking. 4) the UI in treeherder will look quite messy when it comes to chunking- although I predict a small number of chunks 5) this relies upon mozharness to run each possible test suite all the time- sadly this doesn't include all test types, specifically talos and web-platform-tests; we will need bigger changes to mozharness or separate jobs. 6) the treeherder symbols/codes are hacky- we need a better process for mapping these 7) the data in tree needs to be validated with each component owner 8) I imagine android, coverage, stylo-disabled, etc. will present interesting challenges 9) until we get off buildbot, we cannot do reftests on windows 10, nor talos 10) currently we would run all the jsreftests on every commit- ignoring the logic added to taskcluster to run when certain files change With all this said, there are probably other things I have overlooked- with some time on this we can start getting a solution that is less hacky and ready for production.
You need to log in before you can comment on or make changes to this bug.