Open Bug 982674 Opened 6 years ago Updated 5 years ago

Support subtests in manifest parser

Categories

(Testing :: Mozbase, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

People

(Reporter: jgraham, Unassigned)

Details

No description provided.
(sorry bugzilla UI fail)

I have a need for "manifests" that support subtests, so that the expected results of individual subtests can be recorded. At present manifestdestiny doesn't support this use case. 

An example of my solution is at [1] which uses indentation for hierarchy. It's not super-clear to me how to do this in a way that looks exactly like an ini file, since they are non-hierarchical by design.

[1] http://hg.mozilla.org/projects/cedar/file/9af41dfd75f0/testing/web-platform-tests/metadata/dom/historical.html.ini
This seems hard :/

Do you have an idea how often we would run into cases where we'd want to allow subtests to fail, but still have the overall test pass? There's a pretty big assumption throughout our automation at the moment that TEST-PASS implies all subtests pass.
Well in web-platform-tests it happens all the time. For example [1], [2]. It's considered totally normal in that testsuite to put many tests in a file, and totally normal to have tests that don't pass in specific implementations. Throwing away whole files because one subtest isn't passing would dramatically reduce the usefulness of the suite.

[1] http://w3c-test.org/dom/interfaces.html
[2] http://w3c-test.org/dom/historical.html
Summary: Support subtests in manifest destiny → Support subtests in manifest parser
Hm, given that subtests are all part of the same file, we could put them all under the section like:

[file.js]
test1
test2

But here the problem is that you cannot individually disable tests, or specify other keys for them. So why can't we make it similar to include or the upcoming parent section?

[file.js:test1]
disabled = ...

[file.js:test2]
skip-if = ...
You need to log in before you can comment on or make changes to this bug.