Last Comment Bug 801898 - Get GaiaTest UI smoke tests running reliably in Jenkins CI
: Get GaiaTest UI smoke tests running reliably in Jenkins CI
Status: RESOLVED WONTFIX
:
Product: Testing
Classification: Components
Component: General (show other bugs)
: unspecified
: All Gonk (Firefox OS)
: -- normal (vote)
: B2G C1 (to 19nov)
Assigned To: Mozilla Web QA Team (:mozwebqa)
:
Mentors:
Depends on: 853622 860854 865232 924912 971747 1011358 754220 779011 804515 804549 805475 806988 809959 811053 811685 811949 812521 813561 813583 813934 814037 814731 816057 816514 816641 816643 819351 819809 821084 821766 827783 828109 829566 830197 830240 831852 832045 832563 833061 833761 833764 834293 837394 838158 838786 839675 840612 840931 843793 844955 845925 846456 847655 849183 852985 853878 855029 857717 859264 859748 860849 861193 862053 862156 862167 863117 863608 864296 864725 866342 867220 867373 867573 867672 869041 869198 869688 874599 874822 874961 875442 875474 875478 876411 876695 876697 876704 877014 877163 877174 877397 878133 883279 886844 887274 888284 914477 921194 921620 922467 923173 927404 932385 932804 937053 943018 943439 946130 947171 947317 947881 948852 949401 956761 959520 965256 972557 975169 987760 990580 991262 991499 997791 999506 1000617 1011668 1016821 1020910 1020974
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-15 15:20 PDT by Jonathan Griffin (:jgriffin)
Modified: 2016-05-27 10:32 PDT (History)
16 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Jonathan Griffin (:jgriffin) 2012-10-15 15:20:30 PDT
Zac now has some Gaia tests at https://github.com/zacc/gaia-ui-tests.  We should setup a job to get these running at http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/.
Comment 1 Jonathan Griffin (:jgriffin) 2012-10-15 16:04:53 PDT
Before we can do this, we need to figure out a better mechanism for getting patches to the harness in the tree (https://github.com/mozilla-b2g/gaia/pull/5750), or move the harness elsewhere.
Comment 2 Jonathan Griffin (:jgriffin) 2012-10-16 16:56:43 PDT
I tried getting this running on the emulator on CI, with very bad results!

http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/console

The problem is that the emulator is slow, and gaia on the emulator is even slower.  A transition that takes 1s on a device can be 20+s on the emulator, and during that period the app is in an indeterminate state.  I think it will be hard to modify the harness and the tests in such a way that they don't generate random timeouts, so I'm going to try and get these going on the b2g desktop build instead.

The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old job for them which hasn't been run in 2 months, so I'll resurrect that and see where we get.
Comment 3 Zac C (:zac) 2012-10-17 01:02:27 PDT
I haven't run on the emulator yet but even on my device I am finding it a bit flaky and I wouldn't call it ready for a CI run yet!
Comment 4 Malini Das [:mdas] - Away, not checking bugmail 2012-10-17 06:54:51 PDT
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> I tried getting this running on the emulator on CI, with very bad results!
> 
> http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/
> console
> 
> The problem is that the emulator is slow, and gaia on the emulator is even
> slower.  A transition that takes 1s on a device can be 20+s on the emulator,
> and during that period the app is in an indeterminate state.  I think it
> will be hard to modify the harness and the tests in such a way that they
> don't generate random timeouts, so I'm going to try and get these going on
> the b2g desktop build instead.
> 
> The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old
> job for them which hasn't been run in 2 months, so I'll resurrect that and
> see where we get.

Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/? Granted,  I don't know which one of these categories to choose... I've been testing the gaia integration tests with the nightly builds and it works well.
Comment 5 Malini Das [:mdas] - Away, not checking bugmail 2012-10-17 08:07:53 PDT
(In reply to Malini Das [:mdas] from comment #4)
> Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/?
> Granted,  I don't know which one of these categories to choose... I've been
> testing the gaia integration tests with the nightly builds and it works well.

Actually, I don't know what the images in the cedar* folders are, but they don't look like b2g-desktop ones, and I'm not sure what's going on for ash. At least we get mozilla-central for now.
Comment 6 Jonathan Griffin (:jgriffin) 2012-10-17 09:46:14 PDT
I didn't originally consider the releng builds because they're only generated once-per-day, not per-commit.  However, for these tests, that's probably OK, since they're much more sensitive to gaia changes than gecko ones.
Comment 7 Jonathan Griffin (:jgriffin) 2012-10-22 14:43:45 PDT
So, it currently looks like using b2g desktop builds is not a workable approach, due to severe stability issues with that build (see https://bugzilla.mozilla.org/buglist.cgi?field0-0-0=status_whiteboard;classification=Client%20Software;classification=Components;query_format=advanced;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=READY;bug_status=ASSIGNED;bug_status=REOPENED;type0-0-0=substring;value0-0-0=b2g-desktop-builds;product=Boot2Gecko;product=Core;list_id=4746594).

Emulators are too slow; in order to get Gaia tests running smoothly on them it would probably take a lot of tweaking of both the tests and the harness.

That leaves us with devices.  Malini is going to try and get them running on a panda.  If she's successful, she may be able to run them 1x/day, reporting to autolog, until we roll out large-scale panda automation.

It's also possible we could attach an otoro or unagi to autophone, and run them with a similar frequency there, but in my experience, otoros (and thus likely unagis) are poor automation candidates, since (at least for me) they frequency encounter problems during flashing which can't be automatically recovered.
Comment 8 Jonathan Griffin (:jgriffin) 2012-10-31 17:40:17 PDT
I'm unassigning myself from this since it's being handled by stephen's team.
Comment 9 Dave Hunt (:davehunt) 2012-12-07 07:07:46 PST
Updating summary as these are now running, but we are having reliability issues. Bugs related to reliability are listed as dependencies. Also limited this to UI smoke tests as we have dependant bugs listed that are blocking these from being completed.

Note You need to log in before you can comment on or make changes to this bug.