Get functional tests running in the github repository

RESOLVED FIXED

Status

P1
normal
Rank:
10
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: standard8, Assigned: standard8)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

User Story

- Functional tests should be able to be run as part of make test (but turn-offable for Travis).
- Functional tests should assume the dev server or a local server.
- Functional tests should be able to be run against either built/ or dist.

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
We need to get the functional tests running in the new github repository.

For now, we should keep using Marionette, and set it up to run locally. We probably don't want to run on travis at the moment (due to external networking), but we can keep the tests running on Nils' machine.

The important thing is to get them runnable again in the github repo.

I think what we can do is:

- Base our scripts on those in https://github.com/nils-ohlmeier/hello-e2e-tests
- Don't download anything
- Update bin/require.pip with the marionette dependencies (and specific versions) that get installed via https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/setup.sh#L75

Note: I'm assuming these are all available via pip.

- Set up a makefile command to run the equivalent of the one in https://github.com/nils-ohlmeier/hello-e2e-tests/blob/50fb3d306acb14dd75dee5a561e6fa6b5dc37f5d/runtests.sh

There may be more complexity to it that I've mentioned above.
(Assignee)

Updated

3 years ago
Rank: 18
(Assignee)

Updated

3 years ago
Rank: 18 → 14
(Assignee)

Updated

3 years ago
Blocks: 976134
Rank: 14 → 27
Priority: P1 → P2
(Assignee)

Updated

3 years ago
User Story: (updated)
(Assignee)

Comment 1

3 years ago
This is a must-have for the next iteration. We're breaking these too frequently, and we need to be extending them.
Rank: 27 → 5
Priority: P2 → P1
(Assignee)

Updated

3 years ago
Rank: 5 → 10

Updated

3 years ago
Blocks: 1248604

Updated

3 years ago
Assignee: nobody → standard8
(Assignee)

Comment 2

3 years ago
I paired with Dave, and we came up with an in development branch for this bug:

https://github.com/Standard8/loop/tree/bug-1238562-functional

Its all functional, but assumes the executable is Firefox nightly on Mac. We have some ideas to fix this nicely that we'll investigate more tomorrow.
Created attachment 8724052 [details] [review]
[loop] Standard8:bug-1238562-functional > mozilla:master
(Assignee)

Comment 4

3 years ago
Comment on attachment 8724052 [details] [review]
[loop] Standard8:bug-1238562-functional > mozilla:master

Dave, this is basically the same as what we paired on last night, with the addition of:

- getfx.js: a wrapper around the jpm code that finds binaries for us. The default is nightly, but can be overridden by TEST_BROWSER.
- Removal of the top-level run-server.sh - this was a near duplicate of the one in bin/run-server.sh that we don't use, and I forgot to remove it ages ago.
Attachment #8724052 - Flags: review?(dcritchley)
(Assignee)

Comment 5

3 years ago
I've just pushed a couple more changesets:

- The first adds the options around which FF builds to use, the servers and the standalone.
- The second makes functional tests part of `make test` but turns them off for travis for now.

We'll want to consider Travis or something else later on, but for now, lets get the simple stuff running and developers using them.
Comment on attachment 8724052 [details] [review]
[loop] Standard8:bug-1238562-functional > mozilla:master

Looks Good
Attachment #8724052 - Flags: review?(dcritchley) → review+
(Assignee)

Comment 7

3 years ago
https://github.com/mozilla/loop/commit/3037146602e43d8db391d690453ff58b3f12a436
https://github.com/mozilla/loop/commit/321de9f7309180ff57bab1f487cb69c5b7615460
https://github.com/mozilla/loop/commit/790435fb35d35d41ae5dead26a22ced9b5b72d86

There was also a follow-up to fix the installation of the add-on, I'd made it temporary (as a precursor to things we need elsewhere), but forgotten that we then restart FF, hence removing the add-on. Simple fix for now is to install it permanently, we can make it temp later as necessary:

https://github.com/mozilla/loop/commit/7cbdd9499021d49353432ece515d9f809887c936
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.