1199720 - Setup builds for b2gdroid

Status: RESOLVED FIXED

(B2GDroid Graveyard :: General, defect)

Description

[[:fabrice] Fabrice Desré]

Attachment: mozconfig-b2droid.patch

Comment 1

[[:fabrice] Fabrice Desré]

Jordan,

Here's a tentative patch to set up a nightly mozconfig for b2gdroid. It's very similar to Firefox for android mozconfigs because we need to build with the android toolchain. What we need to add is a way to get gaia available at $GAIADIR because we package it in the apk. In b2g desktop builds gaia ends up in $topsrcdir/gaia but I have no idea how!

I'd like to get per commit builds and nightlies. Then we'll look at spinning tests.

Comment 2

[Jordan Lund (:jlund)]

(In reply to [:fabrice] Fabrice Desré from comment #1)
> Jordan,
> 
> Here's a tentative patch to set up a nightly mozconfig for b2gdroid. It's
> very similar to Firefox for android mozconfigs because we need to build with
> the android toolchain. What we need to add is a way to get gaia available at
> $GAIADIR because we package it in the apk. In b2g desktop builds gaia ends
> up in $topsrcdir/gaia but I have no idea how!
> 
> I'd like to get per commit builds and nightlies. Then we'll look at spinning
> tests.

okay, so it sounds like we need to adopt parts of b2g desktop builds into android builds. at least checking out gaia anyway. we can probably just use Taskcluster (buildbot replacement) to schedule these (since we don't need windows or osx) and we can extend the mozharness script (lives in gecko tree) that does android builds to checkout gaia.

I'll talk to my mgmt about this on wed to see if it's something we can pick up right way. if not, I could help you self serve this. Sound good?

/me keeps needinfo as self reminder to touch base on wed.

Comment 3

[[:fabrice] Fabrice Desré]

Sounds good, thanks Jordan!

Comment 4

[Jordan Lund (:jlund)]

> /me keeps needinfo as self reminder to touch base on wed.

I'm going to investigate the continuous integration work needed for this. do you have any hard timelines for this bug?

Comment 5

[[:fabrice] Fabrice Desré]

(In reply to Jordan Lund (:jlund) from comment #4)
> > /me keeps needinfo as self reminder to touch base on wed.
> 
> I'm going to investigate the continuous integration work needed for this. do
> you have any hard timelines for this bug?

Really asap. I need to have the whole builds + updates set up in a couple of weeks.

Comment 6

[[:fabrice] Fabrice Desré]

Attachment: mozconfig-b2droid.patch

Small update, adding MOZTTDIR since we need it for bug 1201712

Comment 7

[Jordan Lund (:jlund)]

I played around with this today. will have initial patches up tomorrow.

Comment 8

[[:fabrice] Fabrice Desré]

\o/

Comment 9

[Jordan Lund (:jlund)]

similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1163084#c15

I am going through trials and tribulations with getting this setup in taskcluster. I am abandoning effort in pushing new docker images with baked in logic to checkout gaia and instead going to write a new taskcluster build script that sources https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/pull-gaia.sh

hoping to have something complete tomorrow.

Comment 10

[Jordan Lund (:jlund)]

Attachment: 150813_b2gdroid-initial-mozilla-central-2.patch

like https://bugzilla.mozilla.org/show_bug.cgi?id=1163084#c16 I am having an issue self-scheduling this via taskcluster's task graph on try. nevertheless, I think this untested code will get us what we want: scheduling an android variant with a specific mozconfig and gaia checked out beside gecko.

will investigate the graph issue and then get this reviewed.

Comment 11

[[:fabrice] Fabrice Desré]

Jordan, note that we also use MOZTTDIR to get the fonts in the mozconfig:
MOZTTDIR=$topsrcdir/moz-tt

Where moz-tt is a clone of https://github.com/mozilla-b2g/moztt

Comment 12

[Dave Hylands [:dhylands]]

In https://bugzilla.mozilla.org/attachment.cgi?id=8657191 in the file named "nightly" it has a line:

ac_add_options --with-branding=mobile/android/b2gdroid/branding/nightly

but that directory doesn't exist, so maybe its missing from the patch?

Comment 13

[[:fabrice] Fabrice Desré]

Yep, that should be --with-branding=mobile/android/b2gdroid/branding/unofficial

Comment 14

[Jordan Lund (:jlund)]

progress. I can schedule these in TC. the builds themselves are barfing but it's mostly down to a few typos. plan is to submit final patches up for review? tomorrow so we can have per commit abilities.

I'll need to investigate taskcluster state with regards to android nightlies

Comment 15

[sgiles]

Attachment: Setup builds for b2gdroid

Comment 16

[sgiles]

Updated the attachment to point to the 'unofficial' branding rather than the non-existent nightly branding.

Comment 17

[Jordan Lund (:jlund)]

currently working on trying to green up android builds in Taskcluster. all android variants are broken.

current issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1206106#c4

in other news, I've added moz-tt and linked gaia to within the gecko tree as mozconfigs expect it to be inside it. Once 1206106 is resolved, I'll post my patches for review

Comment 18

[Jordan Lund (:jlund)]

Attachment: 150922_b2gdroid-gc-gecko.patch

similar to dummy apk, this patch adds an android variant that:

* schedules the android-b2gdroid variant on try via taskcluster
* checks out gaia
* checks out moz-tt (via tools)
* checks out gecko
* uses a mozconfig from: https://bugzilla.mozilla.org/attachment.cgi?id=8660531

this patch depends heavily on https://bugzilla.mozilla.org/attachment.cgi?id=8664573 which might explain the motivation for making pull-gaia.sh script more generic there.

also like dummy apk variant (https://bugzil.la/1163084), these jobs are broken until android tc builds as a whole are fixed up via https://bugzil.la/1189892 and https://bugzil.la/1206106

I'm just eager to get some eyes on these patches so b2gdroid isn't waiting once the tc android is fixed up.

Comment 19

[Michael Shal [:mshal]]

Comment on attachment 8664577 [details] [diff] [review]
150922_b2gdroid-gc-gecko.patch

>diff --git a/testing/taskcluster/tasks/branches/try/job_flags.yml b/testing/taskcluster/tasks/branches/try/job_flags.yml
>+  android-b2gdroid:
>+    platforms:
>+      - Android
>+    types:
>+      opt:
>+        task: tasks/builds/android_api_11_b2gdroid.yml

I think you'll also need to add 'android-b2gdroid' to testing/taskcluster/tasks/branches/base_job_flags.yml (though tbh I'm not really sure what that file does...)

>diff --git a/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml b/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml
>new file mode 100644
>--- /dev/null
>+++ b/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml
>@@ -0,0 +1,69 @@
>+$inherits:
>+  from: 'tasks/builds/mobile_base.yml'
>+  variables:
>+    build_name: 'android'
>+    build_type: 'opt'

These variables are used by the v2 routes, so you want them to be unique. build_type should probably be 'api-11-b2gdroid-opt', similar to the mozharness config.

Comment 20

[Jordan Lund (:jlund)]

thanks for the review!

I'm going to investigate driving these out of decision task.

Comment 21

[Jordan Lund (:jlund)]

the two dep bugs above unblock us enough to proceed here.

I think supplied mozconfig patch in this bug https://bugzilla.mozilla.org/attachment.cgi?id=8662335 might be out of date.

Currently getting: 21:34:47     INFO -  configure: error: Including platforms/android-* in --with-android-sdk arguments is deprecated.  Use --with-android-sdk=/home/worker/workspace/build/src/android-sdk-linux.

needinfo fabrice

Comment 22

[Jordan Lund (:jlund)]

ah, yeah: https://hg.mozilla.org/mozilla-central/rev/e5ab2152fb00

I'll make that change locally and try again

Comment 23

[Jordan Lund (:jlund)]

Attachment: 150928_b2gdroid-tc-mh.patch

green here: https://treeherder.mozilla.org/logviewer.html#?job_id=12013101&repo=try

notes:
* this patch depends on the mozconfig patch in this bug: https://bugzilla.mozilla.org/attachment.cgi?id=8662335&action=edit and also dummy apk patch in https://bugzilla.mozilla.org/attachment.cgi?id=8667093

Comment 24

[Jordan Lund (:jlund)]

Comment on attachment 8662335 [details] [diff] [review]
Setup builds for b2gdroid

sgiles: I needed to add the following patch: http://people.mozilla.org/~jlund/b2gdroid-mozconfig-interdiff.patch

now that we are ready to schedule these, we will need this mozconfig patch reviewed and landed. I am not the best person for this. You will want someone familiar with the logic either from mobile or else from build sys team.

Comment 25

[[:fabrice] Fabrice Desré]

Jordan, the simplest is create a final patch and to ask :nalexander to review it.

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8667094 [details] [diff] [review]
150928_b2gdroid-tc-mh.patch

Review of attachment 8667094 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/scripts/builder/build-linux.sh
@@ +25,5 @@
>  
>  : WORKSPACE                     ${WORKSPACE:=/home/worker/workspace}
>  
> +# some linux variants, e.g. b2gdroid, require gaia
> +: GAIA_DIR                      ${GAIA_DIR}

Would it make more sense for this to be boolean?  CHECKOUT_GAIA or something like that?  It doesn't seem like the actual directory used is critical, since it's symlinked from the srcdir.

Comment 27

[Jordan Lund (:jlund)]

(In reply to Dustin J. Mitchell [:dustin] from comment #26)
> Comment on attachment 8667094 [details] [diff] [review]

> Would it make more sense for this to be boolean?  CHECKOUT_GAIA or something
> like that?  It doesn't seem like the actual directory used is critical,
> since it's symlinked from the srcdir.

testing this feedback here:
https://hg.mozilla.org/try/rev/2210d1a9e1b9
https://treeherder.mozilla.org/#/jobs?repo=try&revision=20537371857b

Comment 28

[Jordan Lund (:jlund)]

Attachment: 150929_b2gdroid-mozconfig.patch

as per: https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c25

this includes https://bugzilla.mozilla.org/attachment.cgi?id=8662335&action=edit and appends the following interdiff: http://people.mozilla.org/~jlund/b2gdroid-mozconfig-interdiff.patch

Comment 29

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8667377 [details] [diff] [review]
150929_b2gdroid-mozconfig.patch

Review of attachment 8667377 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine, but redirecting to fabrice for final stamp.

::: mobile/android/b2gdroid/branding/unofficial/configure.sh
@@ -1,5 @@
>  # This Source Code Form is subject to the terms of the Mozilla Public
>  # License, v. 2.0. If a copy of the MPL was not distributed with this
>  # file, You can obtain one at http://mozilla.org/MPL/2.0/.
>  
> -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`

Unofficial should keep this, I think.

::: mobile/android/b2gdroid/config/mozconfigs/common
@@ +48,5 @@
> +# Treat warnings as errors in directories with FAIL_ON_WARNINGS.
> +ac_add_options --enable-warnings-as-errors
> +
> +ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-fennec-geoloc-api.key
> +ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk.token

Drop this line.

::: mobile/android/b2gdroid/config/mozconfigs/nightly
@@ +1,1 @@
> +. "$topsrcdir/mobile/android/b2gdroid/config/mozconfigs/common"

Do we need a branding/nightly/configure.sh?

@@ +14,5 @@
> +
> +export MOZILLA_OFFICIAL=1
> +export MOZ_TELEMETRY_REPORTING=1
> +
> +MOZ_ANDROID_GECKOLIBS_AAR=1

Drop this line.

Comment 30

[[:fabrice] Fabrice Desré]

(In reply to Nick Alexander :nalexander from comment #29)

> Looks fine, but redirecting to fabrice for final stamp.

I wrote that patch initially, so I'm not the best reviewer ;)

> > -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`
> 
> Unofficial should keep this, I think.

ok


> Do we need a branding/nightly/configure.sh?

I don't think so. We have an "unofficial" branding, and a nightly mozconfig.

Comment 31

[Jordan Lund (:jlund)]

> > > -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`
> > 
> > Unofficial should keep this, I think.
>

I'll keep this line, drop the others from comment 29

> 
> Do we need a branding/nightly/configure.sh?
> 
> I don't think so. We have an "unofficial" branding, and a nightly mozconfig.

for now we will leave this as is and follow up when things are on try.

Comment 32

[Jordan Lund (:jlund)]

Attachment: 150929_b2gdroid-mozconfig-2.patch

carrying r+

changes to the patch from review feedback in https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c29

http://people.mozilla.org/~jlund/b2gdroid-interdiff-nick-feedback.patch

Comment 33

[Jordan Lund (:jlund)]

I was going to land everything but my latest 'all' push had an orange: https://treeherder.mozilla.org/#/jobs?repo=try&revision=305c99706d37&filter-searchStr=api11

I believe the api11 opt build that is orange is unrelated to my changes so I pulled in latest m-c and re-triggered but my job isn't starting: https://treeherder.mozilla.org/#/jobs?repo=try&revision=381034f42385&exclusion_profile=false

looks like taskcluster is down: https://bugzil.la/1209857

Comment 34

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/de12059b8f36
https://hg.mozilla.org/integration/mozilla-inbound/rev/c6a9741a753c

Comment 35

[Jordan Lund (:jlund)]

Attachment: 150930_b2gdroid-tc-mh.patch

r=dustin

full interdiff: http://people.mozilla.org/~jlund/b2gdroid-tc-interdiff.patch

this patch takes into account: https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c27

and it also adds a 1 liner to fight bitrot:

13:34 <jlund> dustin: would you mind stamping this interdiff: http://people.mozilla.org/~jlund/interdiff
13:34 <jlund> from your orig r+ patch: https://bugzilla.mozilla.org/attachment.cgi?id=8667094
13:35 <jlund> which takes into account the addition of PATH within pull-gaia.sh and the additional use of calling pull-gaia.sh in: http://hg.mozilla.org/integration/mozilla-inbound/rev/964b4851ef52
13:36 <jlund> note, because pull-gaia.sh now messes with the PATH, the pass in of $gaia_props as the third arg isn't really necessary but it's still nice that b2gdroid passes explicit abs paths for its arguments
13:39 <dustin> r+
13:39 <jlund> thanks.

Comment 36

[Jordan Lund (:jlund)]

Attachment: 150930_b2gdroid-mozconfig.patch

r=nalexander

the patch I uploaded in here was wrong: https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c32

but the comment was correct. so this is the same as before but with nick's review feedback

Comment 37

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/de12059b8f36
https://hg.mozilla.org/mozilla-central/rev/c6a9741a753c

Comment 38

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.