Closed Bug 1199720 Opened 4 years ago Closed 4 years ago

Setup builds for b2gdroid

Categories

(B2GDroid Graveyard :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fabrice, Unassigned)

References

Details

Attachments

(2 files, 8 obsolete files)

Attached patch mozconfig-b2droid.patch (obsolete) — Splinter Review
No description provided.
Jordan,

Here's a tentative patch to set up a nightly mozconfig for b2gdroid. It's very similar to Firefox for android mozconfigs because we need to build with the android toolchain. What we need to add is a way to get gaia available at $GAIADIR because we package it in the apk. In b2g desktop builds gaia ends up in $topsrcdir/gaia but I have no idea how!

I'd like to get per commit builds and nightlies. Then we'll look at spinning tests.
Flags: needinfo?(jlund)
(In reply to [:fabrice] Fabrice Desré from comment #1)
> Jordan,
> 
> Here's a tentative patch to set up a nightly mozconfig for b2gdroid. It's
> very similar to Firefox for android mozconfigs because we need to build with
> the android toolchain. What we need to add is a way to get gaia available at
> $GAIADIR because we package it in the apk. In b2g desktop builds gaia ends
> up in $topsrcdir/gaia but I have no idea how!
> 
> I'd like to get per commit builds and nightlies. Then we'll look at spinning
> tests.

okay, so it sounds like we need to adopt parts of b2g desktop builds into android builds. at least checking out gaia anyway. we can probably just use Taskcluster (buildbot replacement) to schedule these (since we don't need windows or osx) and we can extend the mozharness script (lives in gecko tree) that does android builds to checkout gaia.

I'll talk to my mgmt about this on wed to see if it's something we can pick up right way. if not, I could help you self serve this. Sound good?

/me keeps needinfo as self reminder to touch base on wed.
Sounds good, thanks Jordan!
> /me keeps needinfo as self reminder to touch base on wed.

I'm going to investigate the continuous integration work needed for this. do you have any hard timelines for this bug?
Flags: needinfo?(jlund) → needinfo?(fabrice)
(In reply to Jordan Lund (:jlund) from comment #4)
> > /me keeps needinfo as self reminder to touch base on wed.
> 
> I'm going to investigate the continuous integration work needed for this. do
> you have any hard timelines for this bug?

Really asap. I need to have the whole builds + updates set up in a couple of weeks.
Flags: needinfo?(fabrice)
Attached patch mozconfig-b2droid.patch (obsolete) — Splinter Review
Small update, adding MOZTTDIR since we need it for bug 1201712
Attachment #8654211 - Attachment is obsolete: true
I played around with this today. will have initial patches up tomorrow.
\o/
similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1163084#c15

I am going through trials and tribulations with getting this setup in taskcluster. I am abandoning effort in pushing new docker images with baked in logic to checkout gaia and instead going to write a new taskcluster build script that sources https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/pull-gaia.sh

hoping to have something complete tomorrow.
like https://bugzilla.mozilla.org/show_bug.cgi?id=1163084#c16 I am having an issue self-scheduling this via taskcluster's task graph on try. nevertheless, I think this untested code will get us what we want: scheduling an android variant with a specific mozconfig and gaia checked out beside gecko.

will investigate the graph issue and then get this reviewed.
Jordan, note that we also use MOZTTDIR to get the fonts in the mozconfig:
MOZTTDIR=$topsrcdir/moz-tt

Where moz-tt is a clone of https://github.com/mozilla-b2g/moztt
In https://bugzilla.mozilla.org/attachment.cgi?id=8657191 in the file named "nightly" it has a line:

ac_add_options --with-branding=mobile/android/b2gdroid/branding/nightly

but that directory doesn't exist, so maybe its missing from the patch?
Yep, that should be --with-branding=mobile/android/b2gdroid/branding/unofficial
progress. I can schedule these in TC. the builds themselves are barfing but it's mostly down to a few typos. plan is to submit final patches up for review? tomorrow so we can have per commit abilities.

I'll need to investigate taskcluster state with regards to android nightlies
Attached patch Setup builds for b2gdroid (obsolete) — Splinter Review
Attachment #8657191 - Attachment is obsolete: true
Assignee: nobody → sgiles
Status: NEW → ASSIGNED
Updated the attachment to point to the 'unofficial' branding rather than the non-existent nightly branding.
Assignee: sgiles → nobody
Status: ASSIGNED → NEW
Depends on: 1206106
currently working on trying to green up android builds in Taskcluster. all android variants are broken.

current issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1206106#c4

in other news, I've added moz-tt and linked gaia to within the gecko tree as mozconfigs expect it to be inside it. Once 1206106 is resolved, I'll post my patches for review
Attached patch 150922_b2gdroid-gc-gecko.patch (obsolete) — Splinter Review
similar to dummy apk, this patch adds an android variant that:

* schedules the android-b2gdroid variant on try via taskcluster
* checks out gaia
* checks out moz-tt (via tools)
* checks out gecko
* uses a mozconfig from: https://bugzilla.mozilla.org/attachment.cgi?id=8660531

this patch depends heavily on https://bugzilla.mozilla.org/attachment.cgi?id=8664573 which might explain the motivation for making pull-gaia.sh script more generic there.

also like dummy apk variant (https://bugzil.la/1163084), these jobs are broken until android tc builds as a whole are fixed up via https://bugzil.la/1189892 and https://bugzil.la/1206106

I'm just eager to get some eyes on these patches so b2gdroid isn't waiting once the tc android is fixed up.
Attachment #8664577 - Flags: review?(mshal)
Comment on attachment 8664577 [details] [diff] [review]
150922_b2gdroid-gc-gecko.patch

>diff --git a/testing/taskcluster/tasks/branches/try/job_flags.yml b/testing/taskcluster/tasks/branches/try/job_flags.yml
>+  android-b2gdroid:
>+    platforms:
>+      - Android
>+    types:
>+      opt:
>+        task: tasks/builds/android_api_11_b2gdroid.yml

I think you'll also need to add 'android-b2gdroid' to testing/taskcluster/tasks/branches/base_job_flags.yml (though tbh I'm not really sure what that file does...)

>diff --git a/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml b/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml
>new file mode 100644
>--- /dev/null
>+++ b/testing/taskcluster/tasks/builds/android_api_11_b2gdroid.yml
>@@ -0,0 +1,69 @@
>+$inherits:
>+  from: 'tasks/builds/mobile_base.yml'
>+  variables:
>+    build_name: 'android'
>+    build_type: 'opt'

These variables are used by the v2 routes, so you want them to be unique. build_type should probably be 'api-11-b2gdroid-opt', similar to the mozharness config.
Attachment #8664577 - Flags: review?(mshal) → review+
thanks for the review!

I'm going to investigate driving these out of decision task.
Depends on: 1163084
Depends on: 1118394
the two dep bugs above unblock us enough to proceed here.

I think supplied mozconfig patch in this bug https://bugzilla.mozilla.org/attachment.cgi?id=8662335 might be out of date.

Currently getting: 21:34:47     INFO -  configure: error: Including platforms/android-* in --with-android-sdk arguments is deprecated.  Use --with-android-sdk=/home/worker/workspace/build/src/android-sdk-linux.

needinfo fabrice
Flags: needinfo?(fabrice)
ah, yeah: https://hg.mozilla.org/mozilla-central/rev/e5ab2152fb00

I'll make that change locally and try again
Flags: needinfo?(fabrice)
Attached patch 150928_b2gdroid-tc-mh.patch (obsolete) — Splinter Review
green here: https://treeherder.mozilla.org/logviewer.html#?job_id=12013101&repo=try

notes:
* this patch depends on the mozconfig patch in this bug: https://bugzilla.mozilla.org/attachment.cgi?id=8662335&action=edit and also dummy apk patch in https://bugzilla.mozilla.org/attachment.cgi?id=8667093
Attachment #8660531 - Attachment is obsolete: true
Attachment #8664577 - Attachment is obsolete: true
Attachment #8667094 - Flags: review?(dustin)
Comment on attachment 8662335 [details] [diff] [review]
Setup builds for b2gdroid

sgiles: I needed to add the following patch: http://people.mozilla.org/~jlund/b2gdroid-mozconfig-interdiff.patch

now that we are ready to schedule these, we will need this mozconfig patch reviewed and landed. I am not the best person for this. You will want someone familiar with the logic either from mobile or else from build sys team.
Flags: needinfo?(sgiles)
Jordan, the simplest is create a final patch and to ask :nalexander to review it.
Flags: needinfo?(sgiles)
Comment on attachment 8667094 [details] [diff] [review]
150928_b2gdroid-tc-mh.patch

Review of attachment 8667094 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/scripts/builder/build-linux.sh
@@ +25,5 @@
>  
>  : WORKSPACE                     ${WORKSPACE:=/home/worker/workspace}
>  
> +# some linux variants, e.g. b2gdroid, require gaia
> +: GAIA_DIR                      ${GAIA_DIR}

Would it make more sense for this to be boolean?  CHECKOUT_GAIA or something like that?  It doesn't seem like the actual directory used is critical, since it's symlinked from the srcdir.
Attachment #8667094 - Flags: review?(dustin) → review+
(In reply to Dustin J. Mitchell [:dustin] from comment #26)
> Comment on attachment 8667094 [details] [diff] [review]

> Would it make more sense for this to be boolean?  CHECKOUT_GAIA or something
> like that?  It doesn't seem like the actual directory used is critical,
> since it's symlinked from the srcdir.

testing this feedback here:
https://hg.mozilla.org/try/rev/2210d1a9e1b9
https://treeherder.mozilla.org/#/jobs?repo=try&revision=20537371857b
Depends on: 1209614
Comment on attachment 8667377 [details] [diff] [review]
150929_b2gdroid-mozconfig.patch

Review of attachment 8667377 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine, but redirecting to fabrice for final stamp.

::: mobile/android/b2gdroid/branding/unofficial/configure.sh
@@ -1,5 @@
>  # This Source Code Form is subject to the terms of the Mozilla Public
>  # License, v. 2.0. If a copy of the MPL was not distributed with this
>  # file, You can obtain one at http://mozilla.org/MPL/2.0/.
>  
> -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`

Unofficial should keep this, I think.

::: mobile/android/b2gdroid/config/mozconfigs/common
@@ +48,5 @@
> +# Treat warnings as errors in directories with FAIL_ON_WARNINGS.
> +ac_add_options --enable-warnings-as-errors
> +
> +ac_add_options --with-mozilla-api-keyfile=/builds/mozilla-fennec-geoloc-api.key
> +ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk.token

Drop this line.

::: mobile/android/b2gdroid/config/mozconfigs/nightly
@@ +1,1 @@
> +. "$topsrcdir/mobile/android/b2gdroid/config/mozconfigs/common"

Do we need a branding/nightly/configure.sh?

@@ +14,5 @@
> +
> +export MOZILLA_OFFICIAL=1
> +export MOZ_TELEMETRY_REPORTING=1
> +
> +MOZ_ANDROID_GECKOLIBS_AAR=1

Drop this line.
Attachment #8667377 - Flags: review?(nalexander)
Attachment #8667377 - Flags: review?(fabrice)
Attachment #8667377 - Flags: review+
(In reply to Nick Alexander :nalexander from comment #29)

> Looks fine, but redirecting to fabrice for final stamp.

I wrote that patch initially, so I'm not the best reviewer ;)

> > -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`
> 
> Unofficial should keep this, I think.

ok


> Do we need a branding/nightly/configure.sh?

I don't think so. We have an "unofficial" branding, and a nightly mozconfig.
Attachment #8667377 - Flags: review?(fabrice)
> > > -ANDROID_PACKAGE_NAME=org.mozilla.b2gdroid_`echo $USER | sed 's/-/_/g'`
> > 
> > Unofficial should keep this, I think.
>

I'll keep this line, drop the others from comment 29

> 
> Do we need a branding/nightly/configure.sh?
> 
> I don't think so. We have an "unofficial" branding, and a nightly mozconfig.

for now we will leave this as is and follow up when things are on try.
I was going to land everything but my latest 'all' push had an orange: https://treeherder.mozilla.org/#/jobs?repo=try&revision=305c99706d37&filter-searchStr=api11

I believe the api11 opt build that is orange is unrelated to my changes so I pulled in latest m-c and re-triggered but my job isn't starting: https://treeherder.mozilla.org/#/jobs?repo=try&revision=381034f42385&exclusion_profile=false

looks like taskcluster is down: https://bugzil.la/1209857
Keywords: leave-open
r=dustin

full interdiff: http://people.mozilla.org/~jlund/b2gdroid-tc-interdiff.patch

this patch takes into account: https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c27

and it also adds a 1 liner to fight bitrot:

13:34 <jlund> dustin: would you mind stamping this interdiff: http://people.mozilla.org/~jlund/interdiff
13:34 <jlund> from your orig r+ patch: https://bugzilla.mozilla.org/attachment.cgi?id=8667094
13:35 <jlund> which takes into account the addition of PATH within pull-gaia.sh and the additional use of calling pull-gaia.sh in: http://hg.mozilla.org/integration/mozilla-inbound/rev/964b4851ef52
13:36 <jlund> note, because pull-gaia.sh now messes with the PATH, the pass in of $gaia_props as the third arg isn't really necessary but it's still nice that b2gdroid passes explicit abs paths for its arguments
13:39 <dustin> r+
13:39 <jlund> thanks.
Attachment #8667094 - Attachment is obsolete: true
Attachment #8668109 - Flags: review+
r=nalexander

the patch I uploaded in here was wrong: https://bugzilla.mozilla.org/show_bug.cgi?id=1199720#c32

but the comment was correct. so this is the same as before but with nick's review feedback
Attachment #8667377 - Attachment is obsolete: true
Attachment #8667712 - Attachment is obsolete: true
Attachment #8668113 - Flags: review+
Depends on: 1210247
Depends on: 1210631
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.