Closed Bug 1071977 Opened 10 years ago Closed 8 years ago

functional tests should be modularized so it's easy to write more tests

Categories

(Hello (Loop) :: General, defect, P4)

Product:
Hello (Loop)
Component:
General
Platform:
x86
macOS
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: dmosedale, Unassigned)

References

Details

(Whiteboard: [tech-debt])

User Story

As a developer who has the functional tests already configured to run, I should be able to easily add a new test to the functional testing framework.

Acceptance criteria:

* utility functions must be structured to be easily sharable and discoverable (they are currently inside the one test we've got)
* adding a new test should be straightforward, and (if necessary) documented
* the documentation should provide some guidance on how to decide when to add a functional test (perhaps including a link to the "Testing Pyramid" section of http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
In Loop, we want to do what Marionette does in its own test suite, which is group one or a small set of tests per file, since that scales pretty well.  However, we need to do expensive setup and teardown (starting & ending a bunch of servers) that we can't afford to do before and after every test. 

In general, we'll have groups of tests, such as the various tests that only require one browser and no servers, and the various tests that require a bunch of servers, and perhaps others. 

A straw-man proposal for how one might do this with manifest.ini is at <https://etherpad.mozilla.org/manifest-grouping-format>.
Dan -

I'm not as familiar with Marionette python tests as I am with Marionette JS, which allows you to group several tests into a "suite" like this:
https://github.com/mozilla-b2g/gaia/blob/master/apps/email/test/marionette/next_previous_test.js#L101

Are you looking for comparable functionality from Marionette python (i.e. have a handful of tests grouped into a 'suite' that share one Setup and Teardown)?  Actually, I didn't even realize this wasn't available...
Flags: needinfo?(dmose)
(In reply to Richard Pappalardo [:rpapa][:rpappalardo] from comment #1)
>
> Are you looking for comparable functionality from Marionette python (i.e.
> have a handful of tests grouped into a 'suite' that share one Setup and
> Teardown)? 

Yes, almost exactly.  The proposal I mention in the etherpad above is framed to scale across multiple files so that we can specifically avoid ending up with all of our functional tests in one or a few files (as happened with the Talkilla prototype <https://github.com/mozilla/talkilla/blob/master/test/functional/test_MultipleBrowsers.py).
Flags: needinfo?(dmose)
This sort of management is considered out of scope for Marionette itself.  Consider seeing if Whimboo's setup/teardown stuff of the mozmill replacement could be reused.  David thinks the proposal of manifests in the Etherpad link from the user story could be a reasonable start as well.

The manifest_parser module from the a-team may be useful here.
Component: Marionette → General
Product: Testing → Loop
Probably the best bet here is to pull it into the backlog.  Setting firefox-backlog+ in the interest of seeing it triaged soon.  Not sure what sort of priority or rank I should assign, so not touching those for now.
Flags: firefox-backlog+
Whiteboard: [affects=loop] → [affects=loop] [tech-debt]
Note that we may also be able to use Python modules to good effect.  This is effectively a "spike" bug, as we need to figure out what we want to do  before we can actually do it.
Blocks: 1141694
Rank: 45
Priority: -- → P4
Summary: it should be possible for marionette test harness to have separate setup & teardown for different suites → marionette should be modularized so it's easy to write more tests
Keywords: ateam-marionette-runner
Summary: marionette should be modularized so it's easy to write more tests → functional tests should be modularized so it's easy to write more tests
Whiteboard: [affects=loop] [tech-debt] → [tech-debt]
Blocks: 1160677
No longer blocks: 1160677
User Story: (updated)
Blocks: 1160677
User Story: (updated)
Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.