1602377 - mochitest run should fail if a test file is not part of any manifest file (browser.ini, mochitest.ini etc.)

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/263976454789 added a browser-chrome test but didn't add it to any manifest. As a result, it didn't get executed.

There should be a failure message under this condition but the mochitest run the test from the manifest anyway to catch issues with product changes or modified test or tests added to the manifest.

Comment 1

[Andrew Halberstadt [:ahal]]

This should probably be implemented as a linter.

It's not entirely clear how to determine whether a file is a test or not. We'd need to invent heuristics (e.g, any file that starts with browser_ or test_ in a test(s) directory). There could be false positives we'll need to rename.