982674 - Support subtests in manifest parser

Status: NEW

(Testing :: Mozbase, defect)

Comment 1

[James Graham [:jgraham]]

(sorry bugzilla UI fail)

I have a need for "manifests" that support subtests, so that the expected results of individual subtests can be recorded. At present manifestdestiny doesn't support this use case. 

An example of my solution is at [1] which uses indentation for hierarchy. It's not super-clear to me how to do this in a way that looks exactly like an ini file, since they are non-hierarchical by design.

[1] http://hg.mozilla.org/projects/cedar/file/9af41dfd75f0/testing/web-platform-tests/metadata/dom/historical.html.ini

Comment 2

[Andrew Halberstadt [:ahal]]

This seems hard :/

Do you have an idea how often we would run into cases where we'd want to allow subtests to fail, but still have the overall test pass? There's a pretty big assumption throughout our automation at the moment that TEST-PASS implies all subtests pass.

Comment 3

[James Graham [:jgraham]]

Well in web-platform-tests it happens all the time. For example [1], [2]. It's considered totally normal in that testsuite to put many tests in a file, and totally normal to have tests that don't pass in specific implementations. Throwing away whole files because one subtest isn't passing would dramatically reduce the usefulness of the suite.

[1] http://w3c-test.org/dom/interfaces.html
[2] http://w3c-test.org/dom/historical.html

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, given that subtests are all part of the same file, we could put them all under the section like:

[file.js]
test1
test2

But here the problem is that you cannot individually disable tests, or specify other keys for them. So why can't we make it similar to include or the upcoming parent section?

[file.js:test1]
disabled = ...

[file.js:test2]
skip-if = ...