Closed Bug 879976 Opened 9 years ago Closed 9 years ago

Integrate Marionette JS client into gecko

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jgriffin, Assigned: wlach)

References

Details

We'd like to take the Marionette JS client that lightsofapollo has written and integrate it into Marionette's canonical testrunner in gecko.  We want to do this for a number of reasons:

 - we get mozharness/TBPL support nearly for free (some minor mozharness script changes will likely be needed)
 - we get the ability to run tests from a manifest for free (important for TBPL)
 - we get TBPL-compatible logging for free
 - we get the ability to keep the Python client, the JS client, and the server always in-sync and versioned
 - we get the ability to run unit tests of the JS client alongside the existing unit tests for the Python client

Lightsofapollo has a slightly cleaned up version of the JS client at https://github.com/lightsofapollo/marionette_js_client

All the tests which consume the Marionette JS client are in (and will stay in) Gaia, e.g., https://github.com/mozilla-b2g/gaia/tree/master/apps/calendar/test/integration.

The idea is to bring marionette.js and a few other generic, non-B2G-specific files (like those that support an xpcshell-compatible transport) into gecko inside the current testrunner.  When the testrunner encounters a JS test, it will invoke xpcshell (once per test) instead of Python's unittest and allow the marionette.js and friends to be loaded along with whatever glue is needed by the tests (via the require() method).

We'd have to figure out a way to support 'require' generically; currently it's defined in https://github.com/mozilla-b2g/gaia/blob/master/test_apps/test-agent/common/vendor/test-agent/test-agent.js.

Ultimately, we should be able to run Marionette JS tests for any platform (Firefox, B2G, etc), and let the test framework (e.g., Gaia integration tests) define all harness-specific functionality while providing a generic marionette client and transport layer.

We'd probably want to provide an assertion library as well; Gaia integration tests use mocha (https://github.com/mozilla-b2g/gaia/tree/master/test_apps/test-agent/common/vendor/mocha), and it seems reasonable to use that here as well.

Since the integration tests are currently busted and people don't appear to be actively working on them, this is a good time to make this kind of architectural change.

Running the gaia integration tests for gaia developers would remain (approximately) unchanged; we could hook things up to a Gaia makefile target like they are now, and the right things would automatically happen (e.g., pull Marionette from pypi, run the tests...).
I've started work on this. Currently working on getting a marionette client going inside xpcshell. Will move onto trying to actually get reasonable tests working afterwards.
Assignee: nobody → wlachance
Wanted to clear up a few points;

>>We'd have to figure out a way to support 'require' generically; currently it's >>defined in https://github.com/mozilla-b2g/gaia/blob/master/test_apps/test-
>>agent/common/vendor/test-agent/test-agent.js.

We use a similar interface with different implementations based on unit test / performance (or integration test) that implementation is for unit tests only and will not work for the performance tests... 

The gaia code is here: https://github.com/mozilla-b2g/gaia/blob/master/tests/js/xpc.js#L33 but essentially its a subscript loader https://github.com/mozilla-b2g/gaia/blob/master/tests/js/xpc.js#L33

>>We'd probably want to provide an assertion library as well; Gaia integration tests >>use mocha (https://github.com/mozilla-b2g/gaia/tree/master/test_apps/test-
>>agent/common/vendor/mocha), and it seems reasonable to use that here as well.

This is actually the test runner https://github.com/visionmedia/mocha 
(it does not provide any actual assertions we use chai.js for that part)


>>Since the integration tests are currently busted and people don't appear to be >>actively working on them, this is a good time to make this kind of architectural >>change.

We had some integration tests (see any apps test/integration/ folder) that are now defunct. The performance tests however use the same framework and run daily https://datazilla.mozilla.org/b2g/ (any apps test/performance folder). We should make sure those are working smoothly at all times.
The Marionette JS client is now being maintained by the gaia team, and resides in github.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.