807792 - Generate gaia- or full-emulator builds that can be used by emulator tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Jonathan Griffin (:jgriffin)]

Currently emulator-based tests use whatever version of gaia happens to be pre-installed on the static emulator in tooltool that the tests use.

However, it appears that B2G is more sensitive to gaia than I would have guessed, even when not running Gaia. See bug 807478, in which a gecko commit broke tests on the emulator; updating Gaia resolved it.  Having a stale Gaia on the emulator prevented it from booting completely.

In order to prevent hard-to-debug bustages, we need to figure out how to get current gaia installed on the emulator during tests.

One approach would be to do full emulator builds in buildbot (ala the existing panda builds), instead of just taking a gecko build and installing it on a prebuilt emulator.  Another approach would be for rel-eng to create gaia builds periodically (the 'nightly' branch of gaia is generally updated 1x/day), which we could download and install in the emulator at the beginning of our tests.

Comment 1

[Chris AtLee [:catlee]]

whoops, didn't mean to fix this

Comment 2

[Andrew Halberstadt [:ahal]]

Are there any plans to move forward with this soon? We keep hitting problems where we can't update the emulator in tooltool due to bustage which in turn blocks developers working on gecko.

For example bz has a patch that fails because it hits a bug that was fixed in gaia master but wasn't in our tooltool version. Therefore he can't land his patch until we update the tooltool version (or all the tests turn red). However we've been blocked from doing so because a bunch of the gaia-ui tests fail when we do.

Besides this, full stack emulators give:
a) developers an easy place to grab an emulator for testing
b) faster test times (we don't need to install gecko into it every chunk)
c) more stability (how many intermittents are caused by using out of date repos?)
d) easier maintainability (we don't have to constantly update emulator snapshots and the inevitable problems that arise when we do)

Comment 3

[Andrew Halberstadt [:ahal]]

We got bit by this again :(

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=ae43242f72d1

Comment 4

[Jonathan Griffin (:jgriffin)]

Not having full emulator builds is becoming increasingly problematic.  We can't catch regressions due to non-gecko components when they occur, so instead we see them when we attempt to update the emulator weeks after the fact, which makes bisection somewhere between really painful and impossible.  See bug 852111, for a single instance of this.

We really need emulator builds built using the same mechanism that is used for panda and unagi builds.

Comment 5

[Chris AtLee [:catlee]]

Should these be done per-push, or on a nightly basis, or something in between?

The downside with per-push is that the emulator builds are several hundred MB AIUI, so this is a lot of data to be transferring and storing.

If we do nightly or every X hours, then how do we change which emulator is used by the tests? Do we use the latest emulator build? What if that one is broken for some reason?

Comment 6

[Jonathan Griffin (:jgriffin)]

The best solution for a number of reasons would be to generate these per-push.  It will be lot of data transferred/stored...will this be a problem?

If we only generate them periodically, we'll still be in the same situation we are in today, but with a smaller regression window when things go wrong.

Comment 7

[cmtalbert]

Catlee, how much work would it be to generate these per push, or even every 4-6 hours? Does the amount of work differ?  Is this something we can do in early Q2? (That might be a Joduinn question)...

Comment 8

[Chris AtLee [:catlee]]

It's probably easier to do them per-push, because then we can use the emulator that was just built for the tests. Doing them every 4-6 hours or nightly means the tests need some automated way of finding the latest good emulator to use.

We're working on this right now. Hoping to have it ready sometime next week.

Comment 9

[Chris AtLee [:catlee]]

Running into build problems that I haven't been able to figure out yet:

host C: emulator64-arm <= external/qemu/audio/audio.c
ERROR: prebuilts/tools/gcc-sdk/../../gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6/bin/x86_64-linux-gcc only run on 64-bit linux
make: *** [out/host/linux-x86/obj/EXECUTABLES/emulator64-arm_intermediates/audio/audio.o] Error 1

this is running in a 32-bit chroot on a 64-bit machine.

Comment 10

[Andrew Halberstadt [:ahal]]

Should we be building a 64 bit emulator? Afaik, we've only ever done 32 bit builds to date.

Not sure about the build error. Do you have gcc-multilibs and g++-multilibs installed? (see also https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Firefox_OS_build_prerequisites#Requirements_for_Linux)

Comment 11

[Chris AtLee [:catlee]]

yeah, that's the weird thing...I'm not trying to build a 64-bit emulator, but the build system thinks that I am.

I've just done
BRANCH=master ./config.sh emulator
./build.sh

Comment 12

[Jonathan Griffin (:jgriffin)]

An emulator build creates both 64-bit and 32-bit emulator executables:

emulator64-arm
emulator64-mips
emulator64-x86
emulator-arm
emulator-mips
emulator_renderer
emulator-ui
emulator-x86

Comment 13

[Chris AtLee [:catlee]]

so it needs to be run on a 64-bit host then?

Comment 14

[Jonathan Griffin (:jgriffin)]

No, it can run on either.  The 'emulator' binary selects the appropriate emulatorFoo binary to run when invoked.

Comment 15

[Chris AtLee [:catlee]]

I mean the build needs to be done on a 64-bit host.

Comment 16

[cmtalbert]

Where are we on this?

Comment 17

[Chris AtLee [:catlee]]

The deps just got cleared today. I'll be taking a fresh look at this tomorrow.

Comment 18

[Chris AtLee [:catlee]]

Sorry, still haven't had a chance to look at this

Comment 19

[Chris AtLee [:catlee]]

Attachment: gecko configs for emulator

Comment 20

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for emulator

Comment 21

[Chris AtLee [:catlee]]

Attachment: fix mozharness special casing

the b2g build system needs some special values for the emulator, and the original values in b2g_build.py were wrong

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/db7759b03389

Comment 23

[Rail Aliiev [:rail]]

in production

Comment 24

[Phil Ringnalda (:philor)]

Since that'll turn it on for b2g18 and b2g18v1_0_1, don't forget to land the config file there, so it doesn't burn on impact.

And don't forget to tell tbpl what it is before it gets turned on, so it doesn't show up as some strange "Other opt" platform.

Comment 25

[Phil Ringnalda (:philor)]

(And yeah, I did hide them on b2g18*, and because of bug 872114 I'm hiding them on try too.)

Comment 26

[Chris AtLee [:catlee]]

Landed https://hg.mozilla.org/build/buildbot-configs/rev/887ccd4e9a58 to fix up try.

Comment 27

[Hal Wine [:hwine] use NI!]

in production

Comment 28

[Andrew Halberstadt [:ahal]]

So a few comments/questions:

1) These are still listed as 'Other opt'
2) Are these posted anywhere on ftp.m.o yet?
3) Are we ready to retire the ics_armv7a_gecko builds and move all our tests to these?

Comment 29

[Jonathan Griffin (:jgriffin)]

1) I filed bug 872197 for this
3) We'll need to schedule all of our tests on this new build on cedar, and compare against the current build, before we can do the official move, I think.  I'll file a bug for this.

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

catlee, could you fix this up ?
b2g_mozilla-central_emulator_dep
Traceback (most recent call last):
  File "/home/buildduty/buildfaster/buildfaster_report.py", line 338, in <module>
    assert False, props['buildername']
AssertionError: b2g_mozilla-central_emulator_dep