916305 - Add Gaia UI tests to Travis CI

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We need to get the Gaia functional UI tests running within Travis-CI against the desktop builds. This was previously set up in the gaia-ui-tests repository to run against nightly builds, but now that these tests will be in-tree we should set up proper CI for them to get early warnings of failures.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I did have an attempt at this, but didn't get very far. James, could you recommend someone more familiar with the existing Travis scripts to help out with this? Essentially we need the following done:

Install gaiatest and dependencies:
cd tests/python/gaia-ui-tests
python setup.py install

Set environment variables:
export GAIATEST_ACKNOWLEDGED_RISKS=true
export GAIATEST_SKIP_WARNING=true

Run the tests:
gaiatest --app=b2gdesktop --binary=/path/to/b2g-bin --profile=/path/to/profile --type=b2g-xfail --restart gaiatest/tests/manifest.ini

Comment 2

[James Lal [:lightsofapollo]]

[who else can help]

We have a few people I can think of:

[? for they are on the blame but not sure how familiar they are yurenju/timdream wrote most of it]

myself [California]
yurenju: [Taipei]
? julienw: [Paris] 
? Rik [Paris] 

[my suggestions]

Should be easy enough- We have something to download a recent b2g-desktop already:

https://github.com/mozilla-b2g/gaia/blob/master/tools/ci/unit/travis.sh#L41 [this is for firefox use --product b2g for b2g-desktop] [yes this is yet-more duplication from other things]

The above takes into account the operating system which your running on by default so you can write a standalone shell/python script which will:

- download b2g
- set your env variables
- run the tests

Then we can invoke this from the travis CI script... I think this approach makes sense since your team can own the specific details without adding more to the travis main script... In general I would like to clean up the stuff in there so we can run the individual steps locally and keep the main travis script for travis-y things.

Comment 3

[James Lal [:lightsofapollo]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12259

Pointer to Github pull-request

Comment 4

[James Lal [:lightsofapollo]]

Comment on attachment 805775 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12259

Please see the second commit (but feel free to comment in general but the first is just refactoring).

I don't know too much about pip [or virtualenv] so there might be a better way of doing this. Hopefully this is a good start that we can build on top of.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 805775 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12259

This looks great. Could we add something to the README about running this locally? The failing tests looks like it may be a recent regression, and the test more than likely needs updating.

Comment 6

[Jonathan Griffin (:jgriffin)]

We're not using virtualenv here, and instead use sudo to install Python packages to the system's site-packages.  Generally, this should be avoided, but I'm not sure what effect it has on Travis - it could be harmless in that context.

Comment 7

[James Lal [:lightsofapollo]]

Everything in travis is a clean state so it should be OK- I am open to whatever changes make sense though.

Comment 8

[Jonathan Griffin (:jgriffin)]

If this script is only to be used by Travis (which seems like the case), then it doesn't matter.

Comment 9

[James Lal [:lightsofapollo]]

https://github.com/mozilla-b2g/gaia/commit/6be14d615f377c85ba6216800b631562037dfcc2