Do basic startup test for emulator builds

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
5 years ago
6 months ago

People

(Reporter: catlee, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2382] [b2g])

(Reporter)

Description

5 years ago
Before uploading the b2g emulator builds we should run a basic 'does it start' check, and turn the build red if it fails.
(Reporter)

Comment 1

5 years ago
These should run on all emulator builds {ics,jb}-{opt,debug}
Jonathan, Clint, is this something for your team?
Yes, this mostly belongs to us.

One issue we'll have to work out is whether we can even launch the emulator on the build slaves.  For linux builds, the answer to this is almost certainly no; I'm not certain about OS X.  We may have to do this as a regular test job, in which case it only makes sense to do it on configs where we don't plan on running any other tests, like OS X.
Blocks: 916356

Comment 4

5 years ago
Dminor, is this something you could look into? Ideally, we'd just take the existing emulator mozharness script and run one or two web API tests instead of the full suite. That would at least get us off the ground here.

The existing emulator mozharness script is here: http://mxr.mozilla.org/build/source/mozharness/scripts/b2g_emulator_unittest.py
Flags: needinfo?(dminor)

Updated

5 years ago
Status: NEW → ASSIGNED
Flags: needinfo?(dminor)

Updated

5 years ago
Assignee: nobody → dminor

Comment 5

5 years ago
Do the bld-linux64-ec2-XXX instances used for the {ics,jb}-{opt,debug} emulator builds have xvfb or something similar on them so that we can run the emulator directly on the build machine?

If this is the case, I can do something in the makefiles. Otherwise, I can make a cut down test suite to be scheduled for the builds for which we do currently run tests, like jgriffin suggested.

Thanks!
Flags: needinfo?(catlee)
(Reporter)

Comment 6

5 years ago
Yes, we have Xvfb running on :2.
Flags: needinfo?(catlee)

Updated

5 years ago
Depends on: 927589

Comment 7

5 years ago
The emulator will run on the build machines with some modifications to their configuration (since the builder is x86_64, we need to install some ia32 compatibility libs to be able to run the emulator.)

That said, the emulator seems to run unstably on the build machine. The first marionette test passes, the remainder fail, and I've hit segfaults just running the emulator standalone. I'm using a revision that ran green on tbpl (https://tbpl.mozilla.org/?rev=afae5911a1e04d6eb72b231f954fb75806b66d95).

Based upon this, I'm concerned that we will hit a lot of intermittent failures and end up failing builds unnecessarily. This also seems to go against the general direction we've been aiming towards of moving tests (cpp unittests, js jit-tests) out of make check and onto test machines where they can be restarted without having to retrigger a build.

I think jgriffin's suggestion of adding a cutdown test job for builds for which we don't plan on running other tests may end up being the better way ahead. I already have an initial version of a patch for this.

Before I sink any more time into getting things going on a build machine, I wanted to see if running on test machines would be a "good enough" alternative.
Flags: needinfo?(jgriffin)
Flags: needinfo?(catlee)
Yes, I think this is good enough.  We've known that the emulator is pretty particular about its environment; the ubuntu test slaves were configured with this in mind.  It isn't too surprising that the emulator fails to run reliably on the builders.

Maybe there's some work we can do to reduce the setup/teardown time of the basic startup test jobs, like using a smaller alternative to tests.zip.
Flags: needinfo?(jgriffin)

Comment 9

5 years ago
This can definitely be handled by adding a new configuration file which hard codes a few parameters, but it might be a matter of just setting up a new test job with a large chunk size.

For instance, setting mochitest chunk 1 of 1000 results in 70 tests being run, which completes in 30 seconds on my laptop. It is painful to pull down the tests.zip for that, but if we follow through on bug 917999, that should get better.
Yes, that would work.  Eventually, we could replace the small mochitest set with a small gaiatest set, since the latter would be more sensitive to gaia-related emulator breakages, but we're not currently running gaiatest on emulators and a small mochitest chunk would cover most kinds of bustage in the short to medium term.

Comment 11

5 years ago
Unassigning from me as it sounds like buildbot rather than mozharness work at this point.
Assignee: dminor → nobody
(Reporter)

Updated

5 years ago
Flags: needinfo?(catlee)

Updated

4 years ago
Whiteboard: [b2g] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2376] [b2g]

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2376] [b2g] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2382] [b2g]
(Reporter)

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

6 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.