Closed Bug 1323044 Opened 4 years ago Closed 4 years ago

test lint job for mochitest


(Testing :: General, defect)

Not set


(Not tracked)



(Reporter: jmaher, Assigned: gbrown)


(Blocks 1 open bug)


to get started with a test lint job for mochitest, there are a few parts:
1) create a taskcluster task that is a test task, but uses the when clause to only run when certain files are changed
2) modify mozharness (as needed) to pass through the test files from the task invocation.  I would like to design for large number of files, so ideally a .json file which is effectively a list of files changed which we pass in as a parameter to the mozharness script.  If we only had a couple files it would be ok on the command line.
3) a modification to the mochitest harness to accept the input of a list of files to run in test-lint mode and for each file run properly.  This will have the added complexity of dealing with identifying which subharness to use (chrome, browser-chrome, devtools, plain, jetpack, a11y, etc.).  On top of that we will need to do various activities inside the harness, at first I see this:
* use --repeat 50
* repeat the session with no repeat, but startup/run single test/shutdown x10
* repeat the test in the scope of the manifest (as it is specified, and another cycle in random mode)

this will all take a lot of time, so I would like to verify that we adjust things as needed.  If needed we can make multiple modes --lint-solo, --lint-random, --lint-??? (a future use would be --lint-rr-chaos)

This job should run on try server and all integration branches.  Future bugs will deal with other harnesses (xpcshell, reftest, marionette, web-platform-tests, etc.)
Can you summarise what a ‘test lint’ job is and how it impacts the Mn harness specifically?  I just want to say upfront that --repeat 50 for Mn would take a significant amount of time.
test-lint is currently defined as a way to verify the test is stable.  As these are new tests or changed tests, I only see a few tests running per commit on average.  The idea here is for mochitest, not marionette, although eventually we would like to get test-lint for other harnesses.

Why I don't have specifics in here is that we need to figure out what is useful w.r.t runtime.  For example, I have found some tests which pass 9 times in a row, but fail on the 10th time.  Would running a single Mn job 50 times take >10 minutes?  I understand running a set of tests many times can take a lot of time, I also plan to reduce the size of some of our larger test manifests so when we run per manifest we can see more realistic runtimes.
Depends on: 1329755
Assignee: nobody → gbrown
I'm going to implement this general idea in a different series of bugs.
Closed: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: test-verify
You need to log in before you can comment on or make changes to this bug.