Closed Bug 1174491 Opened 5 years ago Closed 4 years ago

Unmet Visibility Requirement: documentation for Mn and Mnw

Categories

(Testing :: Marionette, defect, critical)

defect
Not set
critical

Tracking

(firefox41 affected)

RESOLVED INCOMPLETE
Tracking Status
firefox41 --- affected

People

(Reporter: philor, Unassigned)

References

Details

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Has_sufficient_documentation requires:

Has a wiki page with:
* An overview of the test-suite.
* Instructions for running locally.
* How to disable an individual failing test.
* The current owner/who to contact for help.
* The Bugzilla product/component where bugs should be filed (Github issues is not discoverable enough and prevents the use of bug dependencies within the rest of the project).

That wiki page is linked to from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing

Currently Mn links to https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette which seems to mostly be about Marionette the driver rather than Marionette the Marionette test suite, and as a result lacks every single one of those (it does say to file bugs in Testing::Marionette, but given that I was looking at it because I want to disable a Loop test, which certainly shouldn't have a bug filed in the component for the harness, it doesn't even meet that one).

Mnw doesn't have a link, so it certainly doesn't meet any of the requirements for the linked document.
I don't actually know who the owner of either one is, because Catch-22, but at least I know someone who ought to know who the owners are.
Flags: needinfo?(dburns)
(In reply to Phil Ringnalda (:philor) from comment #0)
> https://wiki.mozilla.org/Sheriffing/
> Job_Visibility_Policy#Has_sufficient_documentation requires:
> 
> Has a wiki page with:
> * An overview of the test-suite.
> * Instructions for running locally.
> * How to disable an individual failing test.
> * The current owner/who to contact for help.
> * The Bugzilla product/component where bugs should be filed (Github issues
> is not discoverable enough and prevents the use of bug dependencies within
> the rest of the project).
> 
> That wiki page is linked to from
> https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing
> 
> Currently Mn links to
> https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette which seems
> to mostly be about Marionette the driver rather than Marionette the
> Marionette test suite, and as a result lacks every single one of those (it
> does say to file bugs in Testing::Marionette, but given that I was looking
> at it because I want to disable a Loop test, which certainly shouldn't have
> a bug filed in the component for the harness, it doesn't even meet that one).
> 
> Mnw doesn't have a link, so it certainly doesn't meet any of the
> requirements for the linked document.

Mnw and Loop tests have been overloaded with Marionette tests. While Marionette itself has the details the loop tests and Mnw tests don't. 

I think the right solution, and to simplify the manifests (Mnw has already been done and why we have Mnw). We should do so for Loop now too.

Would that suffice?
Flags: needinfo?(dburns)
+1 to the frustration felt by not knowing who owns what within the Mn test suite these days.

I think it's time to give Marionette the mochitest treatment and start splitting things out. I recognize there's an overhead to doing so, but I think that having a clearer picture of ownership makes it worthwhile.

Something like Mn(F L W) etc...

CCing in some stakeholders for Mn-based things. How does this sound?

And yes, the above is orthogonal to the need for Loop etc to get their docs linked to correct places, but it at least makes it more clear as to where the problems are.
I agree with Ryan that the Mn job is a weird mix of different test suites that are not related to one another for other reasons than sharing the same harness.

This is troublesome for many reasons:

1. Functional tests are more prone to instability and state bleeding over in subsequent tests.  The harness is doing a fairly good job at mitigating potential pitfalls, but compartmentalising further would add more security to running the tests.

2. Splitting up Mn into multiple jobs could potentially provide better options for parallelisation, given that the overhead of scheduling jobs isn’t too great (which is arguably a separate problem if it is).

3. Mn is currently use as a “unit test” suite for the Marionette Gecko library, which is untrue in so many ways.  Subsets of it serve as integration tests for it, but a significant number of tests is not directly related to code in testing/marionette whatsoever.  Having a clearer distinction between which tests are integration tests for Marionette, and which are tests for other features simply relying on the Marionette test harness would create a much clearer distinction.
Depends on: 1177447
Mnw job is no longer run and in general tests are stable that are run. Closing for now and will be raising a meta bug to split microformats/loop/migration into their own jobs.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
(In reply to David Burns :automatedtester from comment #5)
> Closing for now and will be raising a meta bug to split
> microformats/loop/migration into their own jobs.

Please note that this is already covered by bug 1313598.
You need to log in before you can comment on or make changes to this bug.