to get started with a test lint job for mochitest, there are a few parts: 1) create a taskcluster task that is a test task, but uses the when clause to only run when certain files are changed 2) modify mozharness (as needed) to pass through the test files from the task invocation. I would like to design for large number of files, so ideally a .json file which is effectively a list of files changed which we pass in as a parameter to the mozharness script. If we only had a couple files it would be ok on the command line. 3) a modification to the mochitest harness to accept the input of a list of files to run in test-lint mode and for each file run properly. This will have the added complexity of dealing with identifying which subharness to use (chrome, browser-chrome, devtools, plain, jetpack, a11y, etc.). On top of that we will need to do various activities inside the harness, at first I see this: * use --repeat 50 * repeat the session with no repeat, but startup/run single test/shutdown x10 * repeat the test in the scope of the manifest (as it is specified, and another cycle in random mode) this will all take a lot of time, so I would like to verify that we adjust things as needed. If needed we can make multiple modes --lint-solo, --lint-random, --lint-??? (a future use would be --lint-rr-chaos) This job should run on try server and all integration branches. Future bugs will deal with other harnesses (xpcshell, reftest, marionette, web-platform-tests, etc.)
Can you summarise what a ‘test lint’ job is and how it impacts the Mn harness specifically? I just want to say upfront that --repeat 50 for Mn would take a significant amount of time.
test-lint is currently defined as a way to verify the test is stable. As these are new tests or changed tests, I only see a few tests running per commit on average. The idea here is for mochitest, not marionette, although eventually we would like to get test-lint for other harnesses. Why I don't have specifics in here is that we need to figure out what is useful w.r.t runtime. For example, I have found some tests which pass 9 times in a row, but fail on the 10th time. Would running a single Mn job 50 times take >10 minutes? I understand running a set of tests many times can take a lot of time, I also plan to reduce the size of some of our larger test manifests so when we run per manifest we can see more realistic runtimes.
I'm going to implement this general idea in a different series of bugs.