776045 - Update gonk toolchain for B2G builds to ICS

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

We're pretty set on shipping FirefoxOS on an ICS kernel now, but our tinderboxen for FOS are still GB. We need to get this upgraded.

Comment 1

[Chris Cooper [:coop] (he/him)]

What exactly is going to need to change on the releng side? We'll need a new mock environment set up, I'm guessing?

Comment 2

[Philipp von Weitershausen [:philikon]]

Yes, an updated toolchain.

Comment 3

[Philipp von Weitershausen [:philikon]]

aaaaand this just burned the tree [1] because developers weren't aware of the dated toolchain on the tinderboxes. Bumping this to a P1.

[1] https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=02627fecb3e2

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The process works like this (similar to bug 748448 and bug 737153)
* developers generate a new gonk toolchain tarball, upload to people
* RelEng publishes to tooltool server, and generates change to http://hg.mozilla.org/mozilla-central/file/default/b2g/config/tooltool-manifests/releng.manifest
* developers push to try with manifest change and any appropriate patches
* if green push to inbound/central, otherwise iterate

Comment 5

[Philipp von Weitershausen [:philikon]]

(In reply to Nick Thomas [:nthomas] from comment #4)
> The process works like this (similar to bug 748448 and bug 737153)

AFAIUI, this is how it *used* to work. This was based on the old B2G build system which has long been obsoleted. I understand jhford has been working on code drops [1] that will allow us to provide RelEng with an entire source code snapshot of the Gonk toolchain. RelEng would then build Gonk (and later Gecko) much like developers do right now. Is this work being tracked anywhere for the RelEng part? If so, expediting that seems like a good idea to move this forward.

[1] https://github.com/mozilla-b2g/B2G/pull/48

Comment 6

[[e-mail] (:bhearsum)]

According to http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/b2g_config.py#l88, we're still using those manifests. So, they need to be updated, and we need to rename some things in b2g_config.py (basically, s/gb/ics/g).

There may be additional builds in the works, but to get the basic b2g-only + desktop builds updated the above needs to be done.

Catlee should chime in about the additional builds, but he's out until Thursday.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(tweaking summary to match reality)

:kmachulis, :philikon: can you give us specific versions of kernel/tools to be used in the new toolchain?

Comment 8

[Philipp von Weitershausen [:philikon]]

(In reply to John O'Duinn [:joduinn] from comment #7)
> :kmachulis, :philikon: can you give us specific versions of kernel/tools to
> be used in the new toolchain?

If by "versions" you mean a code drop (IOW, a manifest with locked down revisions or alternatively a tarball), then yes.

Comment 9

[Chris AtLee [:catlee]]

Is there any need to keep the old GB toolchain around?

Comment 10

[Chris AtLee [:catlee]]

And yes, we're blocked on getting a new toolchain tarball. Where should I file a bug for that?

Comment 11

[Philipp von Weitershausen [:philikon]]

(In reply to Chris AtLee [:catlee] from comment #10)
> And yes, we're blocked on getting a new toolchain tarball.

As explained in comment 5, as far as I understand how it will work from now on, you'll be getting a *source* tarball from which you'll build everything (including the toolchain). This is how developers have been building B2G from the beginning and how it should be built on the tinderboxes as well.

(I'm mostly channeling others here, so they should feel free to correct me. Just trying to move this bug along. It's important.)

Comment 12

[Chris AtLee [:catlee]]

It would be better to get a tarball with the binaries instead so we don't have to compile the toolchain. My understanding was that we could use one of the toolchains in the prebuilt/ directory?

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

That toolchain is just the compiler toolchain.  It doesn't have a lot of the headers that I understand us to depend on.  In the old B2G build system, we had a "package-toolchain" target that would package up all the needed headers and libraries.  The relevant sections of the makefile are:

https://github.com/andreasgal/B2G/blob/master/Makefile#L670
https://github.com/andreasgal/B2G/blob/master/Makefile#L693

It looks like it'd be easy to capture this logic in the new build system in a toolchain packaging script.  I'm on PTO for the rest of the week, but I'll try to get this script written tonight.  Once that's done someone will need to upload the file to the runtime-binaries server as I don't have access to it any more.  We'll also need a new manifest for mozilla-central to point to the new toolchain tarball.

As for the releng portion, we will need the s/gb/ics/ in the buildbot configs.  I am unsure why gingerbread was baked into the builder names at all, as that adds extra complexity to an upgrade like these where it isn't really needed.

We'd like to switch to a full B2G code-drop based build, but getting an updated toolchain is the quickest path to a working ICS based tinderbox builds.  Doing the full code-drop based build doesn't mean that we'd be building the GCC toolchain, only that we'd be building in a 'real' B2G build environment.

Comment 14

[Chris AtLee [:catlee]]

(In reply to John Ford [:jhford] from comment #13)
> That toolchain is just the compiler toolchain.  It doesn't have a lot of the
> headers that I understand us to depend on.  In the old B2G build system, we
> had a "package-toolchain" target that would package up all the needed
> headers and libraries.  The relevant sections of the makefile are:
> 
> https://github.com/andreasgal/B2G/blob/master/Makefile#L670
> https://github.com/andreasgal/B2G/blob/master/Makefile#L693
> 
> It looks like it'd be easy to capture this logic in the new build system in
> a toolchain packaging script.  I'm on PTO for the rest of the week, but I'll
> try to get this script written tonight.  Once that's done someone will need
> to upload the file to the runtime-binaries server as I don't have access to
> it any more.  We'll also need a new manifest for mozilla-central to point to
> the new toolchain tarball.

How big is it?

> As for the releng portion, we will need the s/gb/ics/ in the buildbot
> configs.  I am unsure why gingerbread was baked into the builder names at
> all, as that adds extra complexity to an upgrade like these where it isn't
> really needed.

At the time we were told that we would most likely be running the GB and ICS builds in parallel. It's better to be more explicit about what the target is.

> We'd like to switch to a full B2G code-drop based build, but getting an
> updated toolchain is the quickest path to a working ICS based tinderbox
> builds.  Doing the full code-drop based build doesn't mean that we'd be
> building the GCC toolchain, only that we'd be building in a 'real' B2G build
> environment.

I don't know what you mean by a code-drop based build or a 'real' B2G build environment.

Comment 15

[Chris AtLee [:catlee]]

John, assigning to you to get the toolchain snapshot created.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

I've had the script to do file copying written for a while, just going through the changes since GB.

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: mozconfigs for ICS

This patch adds the mozconfigs that are needed to use a toolchain based on ICS sources.

Comment 18

[Michael Wu [:mwu]]

(In reply to John Ford [:jhford] from comment #17)
> Created attachment 647718 [details] [diff] [review]
> mozconfigs for ICS
> 
> This patch adds the mozconfigs that are needed to use a toolchain based on
> ICS sources.

Can you add --enable-b2g-camera or make sure it works? We have it disabled by default right now to support GB builds.

Comment 19

[Michael Wu [:mwu]]

Hmm, I guess while we're at it, we might as well also make sure that

ac_add_options --enable-media-plugins
ac_add_options --enable-omx-plugin

also work.

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: add an ICS platform

This depends on the mozconfigs change as well as landing the toolchain generation script, and uploading the toolchain to runtime-binaries and landing a manifest to point to the new toolhchain.  There is also a very small buildbotcustom patch

I don't think it's useful to delete GB until ICS looks good.

Comment 21

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: add ICS to getSupportedPlatforms()

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Michael Wu [:mwu] from comment #18)
> (In reply to John Ford [:jhford] from comment #17)
> > Created attachment 647718 [details] [diff] [review]
> > mozconfigs for ICS
> > 
> > This patch adds the mozconfigs that are needed to use a toolchain based on
> > ICS sources.
> 
> Can you add --enable-b2g-camera or make sure it works? We have it disabled
> by default right now to support GB builds.

(In reply to Michael Wu [:mwu] from comment #19)
> Hmm, I guess while we're at it, we might as well also make sure that
> 
> ac_add_options --enable-media-plugins
> ac_add_options --enable-omx-plugin
> 
> also work.

I'll do that for both.  My guess is that for media and omx, I'm going to have to include stagefright headers, at least.

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 647721 [details] [diff] [review]
add an ICS platform

Review of attachment 647721 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fine, but please hold off on landing until ics.manifest exists, and the packages that it points at. TBPL needs to be updated to support these platforms too. Specifically:
http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/default/js/Config.js#l255 and http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/default/js/Data.js#l504 need ics entries.

Comment 24

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Comment on attachment 647718 [details] [diff] [review]
mozconfigs for ICS

Review of attachment 647718 [details] [diff] [review]:
-----------------------------------------------------------------

Looks like both of these are missing 

ac_add_options --enable-b2g-bt --enable-b2g-ril

Which we need for telephony, bluetooth, wifi... We still need to clean up the name of enable-b2g-ril (bug 749834), but that's not a huge deal.

Comment 25

[Michael Wu [:mwu]]

(In reply to Kyle Machulis [:kmachulis] [:qdot] from comment #24)
> Comment on attachment 647718 [details] [diff] [review]
> mozconfigs for ICS
> 
> Review of attachment 647718 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks like both of these are missing 
> 
> ac_add_options --enable-b2g-bt --enable-b2g-ril
> 
> Which we need for telephony, bluetooth, wifi... We still need to clean up
> the name of enable-b2g-ril (bug 749834), but that's not a huge deal.

Those aren't in https://github.com/mozilla-b2g/gonk-misc/blob/master/default-gecko-config though so they're probably enabled by default.

Comment 26

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Oh. Yeah. They are. Oops. r+.

Comment 27

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Comment on attachment 647718 [details] [diff] [review]
mozconfigs for ICS

Review of attachment 647718 [details] [diff] [review]:
-----------------------------------------------------------------

Looks like both of these are missing 

ac_add_options --enable-b2g-bt --enable-b2g-ril

Which we need for telephony, bluetooth, wifi... We still need to clean up the name of enable-b2g-ril (bug 749834), but that's not a huge deal.

Comment 28

[Philipp von Weitershausen [:philikon]]

(In reply to Michael Wu [:mwu] from comment #25)
> Those aren't in
> https://github.com/mozilla-b2g/gonk-misc/blob/master/default-gecko-config
> though so they're probably enabled by default.

Yes, they're enabled by default for MOZ_WIDGET_GONK [1]. I guess that's also what we're building on the tinderboxes, so we're ok.

https://mxr.mozilla.org/mozilla-central/source/configure.in#2045

Comment 29

[John Ford [:jhford] CET/CEST Berlin Time]

Ok, I fixed the --enable-b2g-camera dependencies

https://github.com/mozilla-b2g/B2G/commit/664c2acace8934bcbbb241204fd537b6cfb83a65

I've uploaded the new toolchain and bootstrap script.

Comment 30

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 647718 [details] [diff] [review]
mozconfigs for ICS

http://hg.mozilla.org/mozilla-central/rev/a25d0900dcda

Comment 31

[John Ford [:jhford] CET/CEST Berlin Time]

I've landed a new manifest for the ICS based toolchain:

http://hg.mozilla.org/mozilla-central/rev/a2cac33cda2a

Comment 32

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: add ics builds to tbpl

This is needed for us to use the ICS toolchain

Comment 33

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 648089 [details] [diff] [review]
add ics builds to tbpl

I learned that philor is on vacation.  Can someone else please review it?

Comment 34

[Matt Brubeck (:mbrubeck)]

Comment on attachment 648089 [details] [diff] [review]
add ics builds to tbpl

Review of attachment 648089 [details] [diff] [review]:
-----------------------------------------------------------------

My only worry here was that "Armv7a ICS debug" would be too wide for the current tinderbox layout, but it seems to fit (at least with the default fonts on my Debian box).

Comment 35

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 648089 [details] [diff] [review]
add ics builds to tbpl

http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/1aab18652eca

Comment 36

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 647721 [details] [diff] [review]
add an ICS platform

http://hg.mozilla.org/build/buildbot-configs/rev/5b04a93c9f3c

Comment 37

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 647722 [details] [diff] [review]
add ICS to getSupportedPlatforms()

http://hg.mozilla.org/build/buildbotcustom/rev/b95286f0d677

Comment 38

[Justin Wood (:Callek)]

This made it into production yesterday afternoon.

Comment 39

[Marshall Culpepper [:marshall_law]]

I've confirmed this is working for my ICS patches on Try. We still need to deprecate / remove the GB builders for those patches to land though (possibly another bug?)

Comment 40

[Chris AtLee [:catlee]]

If you want to disable the gb builders, please file a new bug.

Comment 41

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris AtLee [:catlee] from comment #40)
> If you want to disable the gb builders, please file a new bug.

yep, that was the plan.

I don' think there is anything left to do in this bug.