Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Get GaiaTest UI smoke tests running reliably in Jenkins CI

RESOLVED WONTFIX

Status

Testing
General
RESOLVED WONTFIX
5 years ago
a year ago

People

(Reporter: jgriffin, Assigned: mozwebqa)

Tracking

(Depends on: 4 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Zac now has some Gaia tests at https://github.com/zacc/gaia-ui-tests.  We should setup a job to get these running at http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/.
(Reporter)

Updated

5 years ago
Assignee: nobody → jgriffin
(Reporter)

Comment 1

5 years ago
Before we can do this, we need to figure out a better mechanism for getting patches to the harness in the tree (https://github.com/mozilla-b2g/gaia/pull/5750), or move the harness elsewhere.
(Reporter)

Comment 2

5 years ago
I tried getting this running on the emulator on CI, with very bad results!

http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/console

The problem is that the emulator is slow, and gaia on the emulator is even slower.  A transition that takes 1s on a device can be 20+s on the emulator, and during that period the app is in an indeterminate state.  I think it will be hard to modify the harness and the tests in such a way that they don't generate random timeouts, so I'm going to try and get these going on the b2g desktop build instead.

The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old job for them which hasn't been run in 2 months, so I'll resurrect that and see where we get.

Comment 3

5 years ago
I haven't run on the emulator yet but even on my device I am finding it a bit flaky and I wouldn't call it ready for a CI run yet!
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> I tried getting this running on the emulator on CI, with very bad results!
> 
> http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/job/gaiatest/21/
> console
> 
> The problem is that the emulator is slow, and gaia on the emulator is even
> slower.  A transition that takes 1s on a device can be 20+s on the emulator,
> and during that period the app is in an indeterminate state.  I think it
> will be hard to modify the harness and the tests in such a way that they
> don't generate random timeouts, so I'm going to try and get these going on
> the b2g desktop build instead.
> 
> The Jenkins CI doesn't have any current b2g-desktop builds.  There's an old
> job for them which hasn't been run in 2 months, so I'll resurrect that and
> see where we get.

Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/? Granted,  I don't know which one of these categories to choose... I've been testing the gaia integration tests with the nightly builds and it works well.
(In reply to Malini Das [:mdas] from comment #4)
> Can't we use the ones from ftp://ftp.mozilla.org/pub/b2g/tinderbox-builds/?
> Granted,  I don't know which one of these categories to choose... I've been
> testing the gaia integration tests with the nightly builds and it works well.

Actually, I don't know what the images in the cedar* folders are, but they don't look like b2g-desktop ones, and I'm not sure what's going on for ash. At least we get mozilla-central for now.
(Reporter)

Comment 6

5 years ago
I didn't originally consider the releng builds because they're only generated once-per-day, not per-commit.  However, for these tests, that's probably OK, since they're much more sensitive to gaia changes than gecko ones.
(Reporter)

Comment 7

5 years ago
So, it currently looks like using b2g desktop builds is not a workable approach, due to severe stability issues with that build (see https://bugzilla.mozilla.org/buglist.cgi?field0-0-0=status_whiteboard;classification=Client%20Software;classification=Components;query_format=advanced;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=READY;bug_status=ASSIGNED;bug_status=REOPENED;type0-0-0=substring;value0-0-0=b2g-desktop-builds;product=Boot2Gecko;product=Core;list_id=4746594).

Emulators are too slow; in order to get Gaia tests running smoothly on them it would probably take a lot of tweaking of both the tests and the harness.

That leaves us with devices.  Malini is going to try and get them running on a panda.  If she's successful, she may be able to run them 1x/day, reporting to autolog, until we roll out large-scale panda automation.

It's also possible we could attach an otoro or unagi to autophone, and run them with a similar frequency there, but in my experience, otoros (and thus likely unagis) are poor automation candidates, since (at least for me) they frequency encounter problems during flashing which can't be automatically recovered.
Depends on: 804549
(Reporter)

Comment 8

5 years ago
I'm unassigning myself from this since it's being handled by stephen's team.
Assignee: jgriffin → nobody
Assignee: nobody → mozwebqa
(Reporter)

Updated

5 years ago
Target Milestone: --- → B2G C1 (to 19nov)
(Reporter)

Updated

5 years ago
Depends on: 809959
(Reporter)

Updated

5 years ago
Depends on: 811949, 812521, 812670, 811685
(Reporter)

Updated

5 years ago
Depends on: 811053
(Reporter)

Updated

5 years ago
Depends on: 779011
Depends on: 813934
Depends on: 814037
Depends on: 813583
Depends on: 813561
Depends on: 804515
Depends on: 805475
Depends on: 806988
Depends on: 754220
Depends on: 814731
Depends on: 816057
Depends on: 816514
Depends on: 816641
Updating summary as these are now running, but we are having reliability issues. Bugs related to reliability are listed as dependencies. Also limited this to UI smoke tests as we have dependant bugs listed that are blocking these from being completed.
Summary: Get GaiaTest tests running in Jenkins CI → Get GaiaTest UI smoke tests running reliably in Jenkins CI
Depends on: 819351
Depends on: 819809
(Reporter)

Updated

5 years ago
Depends on: 821084
Depends on: 821766
Depends on: 816643
Depends on: 827783
Depends on: 828109
Depends on: 829566
Depends on: 830240
Depends on: 830197
Depends on: 831852
Depends on: 832045
Depends on: 832563
Depends on: 833061
Depends on: 833761
Depends on: 833764
Depends on: 837394
No longer depends on: 812670
Depends on: 838786
Depends on: 839675
Depends on: 840931
Depends on: 834293
Depends on: 840612
Depends on: 838158
Depends on: 843793
Depends on: 844955
Depends on: 846456
Depends on: 847655
Depends on: 849183
Depends on: 852985
Depends on: 855029
Depends on: 853878
Depends on: 857717
Depends on: 853622
Depends on: 859264
Depends on: 859748
Depends on: 860849
Depends on: 860854
Depends on: 862053
Depends on: 862156
Depends on: 862167
Depends on: 863117
Depends on: 845925
Depends on: 863608
Depends on: 864296
Depends on: 861193
Depends on: 865232
Depends on: 866342
Depends on: 867220
Depends on: 867373
Depends on: 864725

Updated

4 years ago
Depends on: 867573
Depends on: 867672
Depends on: 869041
Depends on: 869198
Depends on: 869688
Depends on: 874599
Depends on: 874822
Depends on: 874961
Depends on: 875442
Depends on: 875474
Depends on: 875478
Depends on: 876411
Depends on: 876695
Depends on: 876697
Depends on: 876704
Depends on: 877014
Depends on: 877163
Depends on: 877174
Depends on: 877397
Depends on: 878133
Depends on: 883279
Depends on: 886844
Depends on: 888284
Depends on: 887274
Depends on: 914477
Depends on: 921194
Depends on: 921620
Depends on: 922467
Depends on: 923173
Depends on: 924912

Updated

4 years ago
Depends on: 925580

Updated

4 years ago
No longer depends on: 925580

Updated

4 years ago
Depends on: 911387

Updated

4 years ago
Depends on: 921205

Updated

4 years ago
Depends on: 915876

Updated

4 years ago
Depends on: 915138

Updated

4 years ago
Depends on: 874897

Updated

4 years ago
Depends on: 838153

Updated

4 years ago
No longer depends on: 838153

Updated

4 years ago
No longer depends on: 874897

Updated

4 years ago
No longer depends on: 911387

Updated

4 years ago
No longer depends on: 915138

Updated

4 years ago
No longer depends on: 915876

Updated

4 years ago
No longer depends on: 921194

Updated

4 years ago
Depends on: 921194

Updated

4 years ago
No longer depends on: 921205
Depends on: 927404
Depends on: 932385
Depends on: 943439
Depends on: 947171
Depends on: 949401
Depends on: 947881
Depends on: 948852
Depends on: 943018
Depends on: 956761
Depends on: 947317
Depends on: 932804
Depends on: 946130
Depends on: 959520

Updated

4 years ago
Depends on: 965256
Depends on: 971747
Depends on: 975169
Depends on: 972557
Depends on: 991499
Depends on: 991262
Depends on: 990580
Depends on: 987760
Depends on: 997791
Depends on: 1000617
Depends on: 999506
Depends on: 1011668
Depends on: 1011358
Depends on: 1020910, 1020974

Updated

3 years ago
Depends on: 937053
Depends on: 1024769

Updated

3 years ago
Depends on: 1016821

Updated

3 years ago
No longer depends on: 1024769
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.