1174206 - Migrate Android x86 builds to TaskCluster and make Tier 2

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Chris Cooper [:coop] (he/him)]

Android x86 builds use the fx_desktop_build.py mozharness script invoked with the custom build variant of x86.

Example command line:

/tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_android_64_builds.py --custom-build-variant-cfg x86 --config balrog/production.py --branch mozilla-inbound --build-pool production


Example build logs:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-x86/1434122155/mozilla-inbound-android-x86-bm74-build1-build1142.txt.gz

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

How is this different from bug 1118394?

Comment 2

[Chris Cooper [:coop] (he/him)]

I only see the Android 4.0 API11+ TC build in try treeherder right now, so I want to make sure API9, x86, and debug variants don't get missed.

If you're using that bug to track moving *all* the variants, that's fine. Feel free to dupe.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I'm assuming (!) those won't be much more work than the single build.  I set up just the one to save load on TC until people are actually looking.

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: bug1174206.patch

wip patches.  They don't work yet due to missing java env on worker.  

https://treeherder.mozilla.org/#/jobs?repo=try&revision=028ec64f5e34c9be6df7214c3a00fe518155663f

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1174206.patch

I added openjdk to the tooltool manifest and it ran green.  Will do some further verification to ensure it is correct.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b10ccb969ddc&selectedJob=22887702

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1174206.diff

diff of unzip -l of the old and new apks.  Not sure if the different manifest names are just because it was run on try

Comment 7

[Kim Moir [:kmoir] ET]

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Not sure if it's worth having a creating a second tooltool manifest for android x86 since it's going away and having openjdk in the one for buildbot doesn't seem to harm anything.

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Review of attachment 8765083 [details] [diff] [review]:
-----------------------------------------------------------------

I think adding on to the existing tooltool manifest is fine.

In your try run, I notice that the tc build had a substantially longer run-time than the buildbot build -- do you know why?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

Comment 10

[Kim Moir [:kmoir] ET]

Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

http://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914

Thanks, I'm investigating the issue wrt longer build times.  I know with some of the other conversions, the problem was that tc builds were running on less powerful instance types (with less RAM). I'm running another try build now to try and will look at the instance types each run with.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6b992274233a
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

Comment 12

[Kim Moir [:kmoir] ET]

Yes, this is the problem.  Buildbot builds currently run on r3.xlarge and the tc worker-type I assigned it is android-api-15 (me.2xlarge) which has less memory.  Going to try to use a different worker type definition or create a new one.  None of the tc worker type definitions currently use r3.xlarge according to https://github.com/taskcluster/aws-provisioner

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

So this caused a problem in production.  The issue was that the inheritence chain of config files does the following:

taskcluster/ci/legacy/tasks/builds/android_api_15.yml inherits from
taskcluster/ci/legacy/tasks/builds/android_x86.yml also inherits from
taskcluster/ci/legacy/tasks/builds/mobile_base.yml which inherits from
taskcluster/ci/legacy/tasks/docker_build.yml

The variables passed from the inherit section in android_api_15.yml are the same as android_x86.yml, 

  variables:
    build_name: 'android'
    build_type: 'opt'

which resolve in docker_build.yml to the routes:

  routes:
    - 'index.gecko.v1.{{project}}.revision.linux.{{head_rev}}.{{build_name}}.{{build_type}}'
    - 'index.gecko.v1.{{project}}.latest.linux.{{build_name}}.{{build_type}}'

which will collide.

Joel helped debug this, and has steps for reproducing this issue here https://pastebin.mozilla.org/8880122 .  This should be a simple matter of changing build_name or build_type, but I'm not sure of the impact of making those changes.  Once I find someone who can do the backout, that's what I'll do

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Joel's completed the backout: http://hg.mozilla.org/integration/mozilla-inbound/rev/73732455d235

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1174206-2.patch

new patch, running try run.  Ironically, I had a different build_name in my patch queue locally but not in the patch I submitted to bugzilla

Comment 16

[Kim Moir [:kmoir] ET]

I'm not sure what the output should be given the fix

Kims-MacBook-Pro:mozilla-central kmoir$  ./mach taskgraph target -p ~/Downloads/parameters.yml 
 0:00.19 Generating full task set
 0:00.56 Starting new HTTPS connection (1): hg.mozilla.org
 0:22.41 Generating full task graph
 0:22.42 Generating target task set
TaskLabel==Htn2LYRFRGGPONQ6Y7aNRw
TaskLabel==dtGSMvrkQ8aPgYfDEwoCrQ


jmaher: how do you tell if mach is generating the right taskgraph? Do the task graphs correspond to something that you can verify in the taskcluster tools?

Comment 17

[Kim Moir [:kmoir] ET]

Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

I also changed the worker type to m3.2xlarge but it seems that linux64-opt was allocated to it even though there are r3.2xlarge in the pool so the build wasn't faster. https://tools.taskcluster.net/aws-provisioner/#opt-linux64/

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

:kmoir, this is all cryptic, but actually you solved it because when the patch that landed was there it was an error printed out while running |mach taskgraph target|, you don't see the error.

I have confidence all is well!

Comment 19

[Geoff Brown [:gbrown]]

Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

Review of attachment 8765924 [details] [diff] [review]:
-----------------------------------------------------------------

You may have it right already, but in light of bug 1283091, maybe double-check the x86 route.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cee0d296a34
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89d45ec9d5aa
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown DONTBUILD

Comment 22

[Kim Moir [:kmoir] ET]

Attachment: bug1174206worker.patch

I changed the workertype back to android15 as it had a better pool of high memory instance types

Comment 23

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1cee0d296a34
https://hg.mozilla.org/mozilla-central/rev/89d45ec9d5aa

Comment 24

[Kim Moir [:kmoir] ET]

I have to add another patch to enable on other branches, this was just on try.

Comment 25

[Kim Moir [:kmoir] ET]

Attachment: bug1174206allbranches.patch

patch to enable on other branches

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4fc457de3a7
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown

Comment 27

[Kim Moir [:kmoir] ET]

This is landed on inbound and I've verified on treeherder However, I'm going to ask in the next mobile meeting if Android x86 is still needed. I recall asking in a meeting about six months ago that Android x86 has very small market numbers compared to armv7. Going to confirm if this is still the case and what the future holds for x86 builds from a product standpoint.

Comment 28

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c4fc457de3a7