1323044 - test lint job for mochitest

Status: RESOLVED DUPLICATE of bug 1357513

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

to get started with a test lint job for mochitest, there are a few parts:
1) create a taskcluster task that is a test task, but uses the when clause to only run when certain files are changed
2) modify mozharness (as needed) to pass through the test files from the task invocation.  I would like to design for large number of files, so ideally a .json file which is effectively a list of files changed which we pass in as a parameter to the mozharness script.  If we only had a couple files it would be ok on the command line.
3) a modification to the mochitest harness to accept the input of a list of files to run in test-lint mode and for each file run properly.  This will have the added complexity of dealing with identifying which subharness to use (chrome, browser-chrome, devtools, plain, jetpack, a11y, etc.).  On top of that we will need to do various activities inside the harness, at first I see this:
* use --repeat 50
* repeat the session with no repeat, but startup/run single test/shutdown x10
* repeat the test in the scope of the manifest (as it is specified, and another cycle in random mode)

this will all take a lot of time, so I would like to verify that we adjust things as needed.  If needed we can make multiple modes --lint-solo, --lint-random, --lint-??? (a future use would be --lint-rr-chaos)

This job should run on try server and all integration branches.  Future bugs will deal with other harnesses (xpcshell, reftest, marionette, web-platform-tests, etc.)

Comment 1

[Andreas Tolfsen ❲:ato❳]

Can you summarise what a ‘test lint’ job is and how it impacts the Mn harness specifically?  I just want to say upfront that --repeat 50 for Mn would take a significant amount of time.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

test-lint is currently defined as a way to verify the test is stable.  As these are new tests or changed tests, I only see a few tests running per commit on average.  The idea here is for mochitest, not marionette, although eventually we would like to get test-lint for other harnesses.

Why I don't have specifics in here is that we need to figure out what is useful w.r.t runtime.  For example, I have found some tests which pass 9 times in a row, but fail on the 10th time.  Would running a single Mn job 50 times take >10 minutes?  I understand running a set of tests many times can take a lot of time, I also plan to reduce the size of some of our larger test manifests so when we run per manifest we can see more realistic runtimes.

Comment 3

[Geoff Brown [:gbrown]]

I'm going to implement this general idea in a different series of bugs.