to get started with a test lint job for mochitest, there are a few parts: 1) create a taskcluster task that is a test task, but uses the when clause to only run when certain files are changed 2) modify mozharness (as needed) to pass through the test files from the task invocation. I would like to design for large number of files, so ideally a .json file which is effectively a list of files changed which we pass in as a parameter to the mozharness script. If we only had a couple files it would be ok on the command line. 3) a modification to the mochitest harness to accept the input of a list of files to run in test-lint mode and for each file run properly. This will have the added complexity of dealing with identifying which subharness to use (chrome, browser-chrome, devtools, plain, jetpack, a11y, etc.). On top of that we will need to do various activities inside the harness, at first I see this: * use --repeat 50 * repeat the session with no repeat, but startup/run single test/shutdown x10 * repeat the test in the scope of the manifest (as it is specified, and another cycle in random mode) this will all take a lot of time, so I would like to verify that we adjust things as needed. If needed we can make multiple modes --lint-solo, --lint-random, --lint-??? (a future use would be --lint-rr-chaos) This job should run on try server and all integration branches. Future bugs will deal with other harnesses (xpcshell, reftest, marionette, web-platform-tests, etc.)
Can you summarise what a ‘test lint’ job is and how it impacts the Mn harness specifically? I just want to say upfront that --repeat 50 for Mn would take a significant amount of time.
test-lint is currently defined as a way to verify the test is stable. As these are new tests or changed tests, I only see a few tests running per commit on average. The idea here is for mochitest, not marionette, although eventually we would like to get test-lint for other harnesses. Why I don't have specifics in here is that we need to figure out what is useful w.r.t runtime. For example, I have found some tests which pass 9 times in a row, but fail on the 10th time. Would running a single Mn job 50 times take >10 minutes? I understand running a set of tests many times can take a lot of time, I also plan to reduce the size of some of our larger test manifests so when we run per manifest we can see more realistic runtimes.