Right now it's not possible to specify a specific test method within a restart module with more than one test methods to be failing. You can only do that at the file level. That means right now you will have to create individual test files if any of your test methods is expected to fail during a restart test. It would be nice if we could specify a specific test in a test module to be failing on purpose.
This is in my opinion something that can and should be done Mozmill-side vs manifestparser side. Manifestparser allows arbitrary key/values and is intended to be universal. While there are some keys appropriate to all testing frameworks, the current canonical unit (after much debate) of the manifest is a file (or URL, though this isn't often used). If we want to reconsider this, we can reopen that discussion, though I would encourage including a proposal on how this would work instead of just as a feature request. However, the individual harnesses can do whatever they want with the manifest. So if you did something like [mytest.js] failing-functions = testSomething testSomethingElse testEtc then Mozmill could include the logic to do something with these I'm not actually sure that the manifest is the best place for this information. To think of this another way, lets say you wanted to run mytest.js in isolation. How do you know what what is supposed to do in this test? IMHO manifests are for - listing what should be run - annotating metadata about the test and not for - knowing innards about the test
I agree this should be done mozmill side and not manifestparser side, because the manifest parser is far larger than just the mozmill test suite. I like Jeff's proposal that the mozmill manifests add attribution into their manifests that mozmill knows how to deal with. That seems far more in line with the architecture of the manifest parser.
There is no need anymore for this request. Restart tests are now in a single file and nearly all old-style tests have been converted meanwhile.