1238562 - Get functional tests running in the github repository

Status: RESOLVED FIXED

(Hello (Loop) :: Client, defect, P1)

Description

[Mark Banner (:standard8)]

We need to get the functional tests running in the new github repository.

For now, we should keep using Marionette, and set it up to run locally. We probably don't want to run on travis at the moment (due to external networking), but we can keep the tests running on Nils' machine.

The important thing is to get them runnable again in the github repo.

I think what we can do is:

- Base our scripts on those in https://github.com/nils-ohlmeier/hello-e2e-tests
- Don't download anything
- Update bin/require.pip with the marionette dependencies (and specific versions) that get installed via https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/setup.sh#L75

Note: I'm assuming these are all available via pip.

- Set up a makefile command to run the equivalent of the one in https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/runtests.sh

There may be more complexity to it that I've mentioned above.

Comment 1

[Mark Banner (:standard8)]

This is a must-have for the next iteration. We're breaking these too frequently, and we need to be extending them.

Comment 2

[Mark Banner (:standard8)]

I paired with Dave, and we came up with an in development branch for this bug:

https://github.com/Standard8/loop/tree/bug-1238562-functional

Its all functional, but assumes the executable is Firefox nightly on Mac. We have some ideas to fix this nicely that we'll investigate more tomorrow.

Comment 3

[GitHub Autolander Bot]

Attachment: [loop] Standard8:bug-1238562-functional > mozilla:master

Comment 4

[Mark Banner (:standard8)]

Comment on attachment 8724052 [details] [review]
[loop] Standard8:bug-1238562-functional > mozilla:master

Dave, this is basically the same as what we paired on last night, with the addition of:

- getfx.js: a wrapper around the jpm code that finds binaries for us. The default is nightly, but can be overridden by TEST_BROWSER.
- Removal of the top-level run-server.sh - this was a near duplicate of the one in bin/run-server.sh that we don't use, and I forgot to remove it ages ago.

Comment 5

[Mark Banner (:standard8)]

I've just pushed a couple more changesets:

- The first adds the options around which FF builds to use, the servers and the standalone.
- The second makes functional tests part of `make test` but turns them off for travis for now.

We'll want to consider Travis or something else later on, but for now, lets get the simple stuff running and developers using them.

Comment 6

[David Critchley (:dcritch)]

Comment on attachment 8724052 [details] [review]
[loop] Standard8:bug-1238562-functional > mozilla:master

Looks Good

Comment 7

[Mark Banner (:standard8)]

https://github.com/mozilla/loop/commit/3037146602e43d8db391d690453ff58b3f12a436
https://github.com/mozilla/loop/commit/321de9f7309180ff57bab1f487cb69c5b7615460
https://github.com/mozilla/loop/commit/790435fb35d35d41ae5dead26a22ced9b5b72d86

There was also a follow-up to fix the installation of the add-on, I'd made it temporary (as a precursor to things we need elsewhere), but forgotten that we then restart FF, hence removing the add-on. Simple fix for now is to install it permanently, we can make it temp later as necessary:

https://github.com/mozilla/loop/commit/7cbdd9499021d49353432ece515d9f809887c936