Closed Bug 807792 Opened 7 years ago Closed 6 years ago

Generate gaia- or full-emulator builds that can be used by emulator tests

Categories

(Release Engineering :: General, defect, P2)

All
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: catlee)

References

Details

(Whiteboard: [b2g][gaia][leave open])

Attachments

(3 files)

Currently emulator-based tests use whatever version of gaia happens to be pre-installed on the static emulator in tooltool that the tests use.

However, it appears that B2G is more sensitive to gaia than I would have guessed, even when not running Gaia. See bug 807478, in which a gecko commit broke tests on the emulator; updating Gaia resolved it.  Having a stale Gaia on the emulator prevented it from booting completely.

In order to prevent hard-to-debug bustages, we need to figure out how to get current gaia installed on the emulator during tests.

One approach would be to do full emulator builds in buildbot (ala the existing panda builds), instead of just taking a gecko build and installing it on a prebuilt emulator.  Another approach would be for rel-eng to create gaia builds periodically (the 'nightly' branch of gaia is generally updated 1x/day), which we could download and install in the emulator at the beginning of our tests.
Status: NEW → RESOLVED
Closed: 7 years ago
Priority: -- → P2
Resolution: --- → FIXED
Whiteboard: [b2g][gaia]
whoops, didn't mean to fix this
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Generate gaia builds that can be used by emulator tests → Generate gaia- or full-emulator builds that can be used by emulator tests
Are there any plans to move forward with this soon? We keep hitting problems where we can't update the emulator in tooltool due to bustage which in turn blocks developers working on gecko.

For example bz has a patch that fails because it hits a bug that was fixed in gaia master but wasn't in our tooltool version. Therefore he can't land his patch until we update the tooltool version (or all the tests turn red). However we've been blocked from doing so because a bunch of the gaia-ui tests fail when we do.

Besides this, full stack emulators give:
a) developers an easy place to grab an emulator for testing
b) faster test times (we don't need to install gecko into it every chunk)
c) more stability (how many intermittents are caused by using out of date repos?)
d) easier maintainability (we don't have to constantly update emulator snapshots and the inevitable problems that arise when we do)
Not having full emulator builds is becoming increasingly problematic.  We can't catch regressions due to non-gecko components when they occur, so instead we see them when we attempt to update the emulator weeks after the fact, which makes bisection somewhere between really painful and impossible.  See bug 852111, for a single instance of this.

We really need emulator builds built using the same mechanism that is used for panda and unagi builds.
Should these be done per-push, or on a nightly basis, or something in between?

The downside with per-push is that the emulator builds are several hundred MB AIUI, so this is a lot of data to be transferring and storing.

If we do nightly or every X hours, then how do we change which emulator is used by the tests? Do we use the latest emulator build? What if that one is broken for some reason?
The best solution for a number of reasons would be to generate these per-push.  It will be lot of data transferred/stored...will this be a problem?

If we only generate them periodically, we'll still be in the same situation we are in today, but with a smaller regression window when things go wrong.
Catlee, how much work would it be to generate these per push, or even every 4-6 hours? Does the amount of work differ?  Is this something we can do in early Q2? (That might be a Joduinn question)...
It's probably easier to do them per-push, because then we can use the emulator that was just built for the tests. Doing them every 4-6 hours or nightly means the tests need some automated way of finding the latest good emulator to use.

We're working on this right now. Hoping to have it ready sometime next week.
Running into build problems that I haven't been able to figure out yet:

host C: emulator64-arm <= external/qemu/audio/audio.c
ERROR: prebuilts/tools/gcc-sdk/../../gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6/bin/x86_64-linux-gcc only run on 64-bit linux
make: *** [out/host/linux-x86/obj/EXECUTABLES/emulator64-arm_intermediates/audio/audio.o] Error 1

this is running in a 32-bit chroot on a 64-bit machine.
Should we be building a 64 bit emulator? Afaik, we've only ever done 32 bit builds to date.

Not sure about the build error. Do you have gcc-multilibs and g++-multilibs installed? (see also https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Firefox_OS_build_prerequisites#Requirements_for_Linux)
yeah, that's the weird thing...I'm not trying to build a 64-bit emulator, but the build system thinks that I am.

I've just done
BRANCH=master ./config.sh emulator
./build.sh
An emulator build creates both 64-bit and 32-bit emulator executables:

emulator64-arm
emulator64-mips
emulator64-x86
emulator-arm
emulator-mips
emulator_renderer
emulator-ui
emulator-x86
so it needs to be run on a 64-bit host then?
No, it can run on either.  The 'emulator' binary selects the appropriate emulatorFoo binary to run when invoked.
I mean the build needs to be done on a 64-bit host.
Depends on: 855003
Depends on: 858853
Where are we on this?
Flags: needinfo?(catlee)
The deps just got cleared today. I'll be taking a fresh look at this tomorrow.
Flags: needinfo?(catlee)
Sorry, still haven't had a chance to look at this
Attachment #745990 - Flags: review?(jhopkins)
Attachment #745992 - Flags: review?(jhopkins)
the b2g build system needs some special values for the emulator, and the original values in b2g_build.py were wrong
Attachment #746568 - Flags: review?(jhopkins)
Attachment #745990 - Flags: review?(jhopkins) → review+
Attachment #745992 - Flags: review?(jhopkins) → review+
Attachment #746568 - Flags: review?(jhopkins) → review+
Attachment #745990 - Flags: checked-in+
Whiteboard: [b2g][gaia] → [b2g][gaia][leaveopen]
Attachment #746568 - Flags: checked-in+
https://hg.mozilla.org/mozilla-central/rev/db7759b03389
Assignee: nobody → catlee
Status: REOPENED → RESOLVED
Closed: 7 years ago6 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [b2g][gaia][leaveopen] → [b2g][gaia][leave open]
Attachment #745992 - Flags: checked-in+
Since that'll turn it on for b2g18 and b2g18v1_0_1, don't forget to land the config file there, so it doesn't burn on impact.

And don't forget to tell tbpl what it is before it gets turned on, so it doesn't show up as some strange "Other opt" platform.
Depends on: 872114
(And yeah, I did hide them on b2g18*, and because of bug 872114 I'm hiding them on try too.)
Depends on: 872197
in production
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
So a few comments/questions:

1) These are still listed as 'Other opt'
2) Are these posted anywhere on ftp.m.o yet?
3) Are we ready to retire the ics_armv7a_gecko builds and move all our tests to these?
1) I filed bug 872197 for this
3) We'll need to schedule all of our tests on this new build on cedar, and compare against the current build, before we can do the official move, I think.  I'll file a bug for this.
Blocks: 872765
catlee, could you fix this up ?
b2g_mozilla-central_emulator_dep
Traceback (most recent call last):
  File "/home/buildduty/buildfaster/buildfaster_report.py", line 338, in <module>
    assert False, props['buildername']
AssertionError: b2g_mozilla-central_emulator_dep
Depends on: 874747
Depends on: 877536
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.