801898 - Get GaiaTest UI smoke tests running reliably in Jenkins CI

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Zac now has some Gaia tests at https://github.com/zacc/gaia-ui-tests.  We should setup a job to get these running at http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/.

Comment 1

[Jonathan Griffin (:jgriffin)]

Before we can do this, we need to figure out a better mechanism for getting patches to the harness in the tree (https://github.com/mozilla-b2g/gaia/pull/5750), or move the harness elsewhere.

Comment 2

[Jonathan Griffin (:jgriffin)]

I tried getting this running on the emulator on CI, with very bad results!

http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/console

The problem is that the emulator is slow, and gaia on the emulator is even slower.  A transition that takes 1s on a device can be 20+s on the emulator, and during that period the app is in an indeterminate state.  I think it will be hard to modify the harness and the tests in such a way that they don't generate random timeouts, so I'm going to try and get these going on the b2g desktop build instead.

The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old job for them which hasn't been run in 2 months, so I'll resurrect that and see where we get.

Comment 3

[Zac C (:zac)]

I haven't run on the emulator yet but even on my device I am finding it a bit flaky and I wouldn't call it ready for a CI run yet!

Comment 4

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> I tried getting this running on the emulator on CI, with very bad results!
> 
> http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/
> console
> 
> The problem is that the emulator is slow, and gaia on the emulator is even
> slower.  A transition that takes 1s on a device can be 20+s on the emulator,
> and during that period the app is in an indeterminate state.  I think it
> will be hard to modify the harness and the tests in such a way that they
> don't generate random timeouts, so I'm going to try and get these going on
> the b2g desktop build instead.
> 
> The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old
> job for them which hasn't been run in 2 months, so I'll resurrect that and
> see where we get.

Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/? Granted,  I don't know which one of these categories to choose... I've been testing the gaia integration tests with the nightly builds and it works well.

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Malini Das [:mdas] from comment #4)
> Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/?
> Granted,  I don't know which one of these categories to choose... I've been
> testing the gaia integration tests with the nightly builds and it works well.

Actually, I don't know what the images in the cedar* folders are, but they don't look like b2g-desktop ones, and I'm not sure what's going on for ash. At least we get mozilla-central for now.

Comment 6

[Jonathan Griffin (:jgriffin)]

I didn't originally consider the releng builds because they're only generated once-per-day, not per-commit.  However, for these tests, that's probably OK, since they're much more sensitive to gaia changes than gecko ones.

Comment 7

[Jonathan Griffin (:jgriffin)]

So, it currently looks like using b2g desktop builds is not a workable approach, due to severe stability issues with that build (see https://bugzilla.mozilla.org/buglist.cgi?field0-0-0=status_whiteboard;classification=Client%20Software;classification=Components;query_format=advanced;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=READY;bug_status=ASSIGNED;bug_status=REOPENED;type0-0-0=substring;value0-0-0=b2g-desktop-builds;product=Boot2Gecko;product=Core;list_id=4746594).

Emulators are too slow; in order to get Gaia tests running smoothly on them it would probably take a lot of tweaking of both the tests and the harness.

That leaves us with devices.  Malini is going to try and get them running on a panda.  If she's successful, she may be able to run them 1x/day, reporting to autolog, until we roll out large-scale panda automation.

It's also possible we could attach an otoro or unagi to autophone, and run them with a similar frequency there, but in my experience, otoros (and thus likely unagis) are poor automation candidates, since (at least for me) they frequency encounter problems during flashing which can't be automatically recovered.

Comment 8

[Jonathan Griffin (:jgriffin)]

I'm unassigning myself from this since it's being handled by stephen's team.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Updating summary as these are now running, but we are having reliability issues. Bugs related to reliability are listed as dependencies. Also limited this to UI smoke tests as we have dependant bugs listed that are blocking these from being completed.