If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Enhance Manifestparser to also use test methods in a module as canonical unit

RESOLVED INVALID

Status

Testing Graveyard
Mozmill
--
enhancement
RESOLVED INVALID
5 years ago
a year ago

People

(Reporter: whimboo, Unassigned)

Tracking

Details

(Reporter)

Description

5 years ago
Right now it's not possible to specify a specific test method within a restart module with more than one test methods to be failing. You can only do that at the file level. 

That means right now you will have to create individual test files if any of your test methods is expected to fail during a restart test.

It would be nice if we could specify a specific test in a test module to be failing on purpose.

Comment 1

5 years ago
This is in my opinion something that can and should be done Mozmill-side vs manifestparser side.  Manifestparser allows arbitrary key/values and is intended to be universal.  While there are some keys appropriate to all testing frameworks, the current canonical unit (after much debate) of the manifest is a file (or URL, though this isn't often used).

If we want to reconsider this, we can reopen that discussion, though I would encourage including a proposal on how this would work instead of just as a feature request.

However, the individual harnesses can do whatever they want with the manifest.  So if you did something like

[mytest.js]
failing-functions = testSomething testSomethingElse testEtc

then Mozmill could include the logic to do something with these

I'm not actually sure that the manifest is the best place for this information.  To think of this another way, lets say you wanted to run mytest.js in isolation.  How do you know what what is supposed to do in this test?  IMHO manifests are for

- listing what should be run
- annotating metadata about the test 

and not for 

- knowing innards about the test

Comment 2

5 years ago
I agree this should be done mozmill side and not manifestparser side, because the manifest parser is far larger than just the mozmill test suite.  I like Jeff's proposal that the mozmill manifests add attribution into their manifests that mozmill knows how to deal with. That seems far more in line with the architecture of the manifest parser.
(Reporter)

Comment 3

3 years ago
There is no need anymore for this request. Restart tests are now in a single file and nearly all old-style tests have been converted meanwhile.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INVALID
(Assignee)

Updated

a year ago
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.