- Functional tests should be able to be run as part of make test (but turn-offable for Travis). - Functional tests should assume the dev server or a local server. - Functional tests should be able to be run against either built/ or dist.
We need to get the functional tests running in the new github repository. For now, we should keep using Marionette, and set it up to run locally. We probably don't want to run on travis at the moment (due to external networking), but we can keep the tests running on Nils' machine. The important thing is to get them runnable again in the github repo. I think what we can do is: - Base our scripts on those in https://github.com/nils-ohlmeier/hello-e2e-tests - Don't download anything - Update bin/require.pip with the marionette dependencies (and specific versions) that get installed via https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/setup.sh#L75 Note: I'm assuming these are all available via pip. - Set up a makefile command to run the equivalent of the one in https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/runtests.sh There may be more complexity to it that I've mentioned above.
This is a must-have for the next iteration. We're breaking these too frequently, and we need to be extending them.
Rank: 27 → 5
Priority: P2 → P1
I paired with Dave, and we came up with an in development branch for this bug: https://github.com/Standard8/loop/tree/bug-1238562-functional Its all functional, but assumes the executable is Firefox nightly on Mac. We have some ideas to fix this nicely that we'll investigate more tomorrow.
Created attachment 8724052 [details] [review] [loop] Standard8:bug-1238562-functional > mozilla:master
Comment on attachment 8724052 [details] [review] [loop] Standard8:bug-1238562-functional > mozilla:master Dave, this is basically the same as what we paired on last night, with the addition of: - getfx.js: a wrapper around the jpm code that finds binaries for us. The default is nightly, but can be overridden by TEST_BROWSER. - Removal of the top-level run-server.sh - this was a near duplicate of the one in bin/run-server.sh that we don't use, and I forgot to remove it ages ago.
Attachment #8724052 - Flags: review?(dcritchley)
I've just pushed a couple more changesets: - The first adds the options around which FF builds to use, the servers and the standalone. - The second makes functional tests part of `make test` but turns them off for travis for now. We'll want to consider Travis or something else later on, but for now, lets get the simple stuff running and developers using them.
Comment on attachment 8724052 [details] [review] [loop] Standard8:bug-1238562-functional > mozilla:master Looks Good
Attachment #8724052 - Flags: review?(dcritchley) → review+
https://github.com/mozilla/loop/commit/3037146602e43d8db391d690453ff58b3f12a436 https://github.com/mozilla/loop/commit/321de9f7309180ff57bab1f487cb69c5b7615460 https://github.com/mozilla/loop/commit/790435fb35d35d41ae5dead26a22ced9b5b72d86 There was also a follow-up to fix the installation of the add-on, I'd made it temporary (as a precursor to things we need elsewhere), but forgotten that we then restart FF, hence removing the add-on. Simple fix for now is to install it permanently, we can make it temp later as necessary: https://github.com/mozilla/loop/commit/7cbdd9499021d49353432ece515d9f809887c936
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.