Closed Bug 1174206 Opened 7 years ago Closed 6 years ago

Migrate Android x86 builds to TaskCluster and make Tier 2

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: coop, Assigned: kmoir)

References

Details

(Whiteboard: [taskcluster][mobile][build])

Attachments

(5 files, 1 obsolete file)

Android x86 builds use the fx_desktop_build.py mozharness script invoked with the custom build variant of x86.

Example command line:

/tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_android_64_builds.py --custom-build-variant-cfg x86 --config balrog/production.py --branch mozilla-inbound --build-pool production


Example build logs:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-x86/1434122155/mozilla-inbound-android-x86-bm74-build1-build1142.txt.gz
How is this different from bug 1118394?
I only see the Android 4.0 API11+ TC build in try treeherder right now, so I want to make sure API9, x86, and debug variants don't get missed.

If you're using that bug to track moving *all* the variants, that's fine. Feel free to dupe.
Blocks: 1118394
I'm assuming (!) those won't be much more work than the single build.  I set up just the one to save load on TC until people are actually looking.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1118394
Status: RESOLVED → REOPENED
Component: General Automation → Task Configuration
Product: Release Engineering → Taskcluster
QA Contact: catlee
Resolution: DUPLICATE → ---
Assignee: nobody → kmoir
Summary: Migrate Android x86 builds to TaskCluster → Migrate Android x86 builds to TaskCluster and make Tier 2
Blocks: 1280440
Attached patch bug1174206.patch (obsolete) — Splinter Review
wip patches.  They don't work yet due to missing java env on worker.  

https://treeherder.mozilla.org/#/jobs?repo=try&revision=028ec64f5e34c9be6df7214c3a00fe518155663f
Attached patch bug1174206.patchSplinter Review
I added openjdk to the tooltool manifest and it ran green.  Will do some further verification to ensure it is correct.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b10ccb969ddc&selectedJob=22887702
Attachment #8764739 - Attachment is obsolete: true
Attached file bug1174206.diff
diff of unzip -l of the old and new apks.  Not sure if the different manifest names are just because it was run on try
Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Not sure if it's worth having a creating a second tooltool manifest for android x86 since it's going away and having openjdk in the one for buildbot doesn't seem to harm anything.
Attachment #8765083 - Flags: review?(gbrown)
Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

Review of attachment 8765083 [details] [diff] [review]:
-----------------------------------------------------------------

I think adding on to the existing tooltool manifest is fine.

In your try run, I notice that the tc build had a substantially longer run-time than the buildbot build -- do you know why?
Attachment #8765083 - Flags: review?(gbrown) → review+
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown
Comment on attachment 8765083 [details] [diff] [review]
bug1174206.patch

http://hg.mozilla.org/integration/mozilla-inbound/rev/3e53a0aae914

Thanks, I'm investigating the issue wrt longer build times.  I know with some of the other conversions, the problem was that tc builds were running on less powerful instance types (with less RAM). I'm running another try build now to try and will look at the instance types each run with.
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6b992274233a
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown
Yes, this is the problem.  Buildbot builds currently run on r3.xlarge and the tc worker-type I assigned it is android-api-15 (me.2xlarge) which has less memory.  Going to try to use a different worker type definition or create a new one.  None of the tc worker type definitions currently use r3.xlarge according to https://github.com/taskcluster/aws-provisioner
So this caused a problem in production.  The issue was that the inheritence chain of config files does the following:

taskcluster/ci/legacy/tasks/builds/android_api_15.yml inherits from
taskcluster/ci/legacy/tasks/builds/android_x86.yml also inherits from
taskcluster/ci/legacy/tasks/builds/mobile_base.yml which inherits from
taskcluster/ci/legacy/tasks/docker_build.yml

The variables passed from the inherit section in android_api_15.yml are the same as android_x86.yml, 

  variables:
    build_name: 'android'
    build_type: 'opt'

which resolve in docker_build.yml to the routes:

  routes:
    - 'index.gecko.v1.{{project}}.revision.linux.{{head_rev}}.{{build_name}}.{{build_type}}'
    - 'index.gecko.v1.{{project}}.latest.linux.{{build_name}}.{{build_type}}'

which will collide.

Joel helped debug this, and has steps for reproducing this issue here https://pastebin.mozilla.org/8880122 .  This should be a simple matter of changing build_name or build_type, but I'm not sure of the impact of making those changes.  Once I find someone who can do the backout, that's what I'll do
new patch, running try run.  Ironically, I had a different build_name in my patch queue locally but not in the patch I submitted to bugzilla
I'm not sure what the output should be given the fix

Kims-MacBook-Pro:mozilla-central kmoir$  ./mach taskgraph target -p ~/Downloads/parameters.yml 
 0:00.19 Generating full task set
 0:00.56 Starting new HTTPS connection (1): hg.mozilla.org
 0:22.41 Generating full task graph
 0:22.42 Generating target task set
TaskLabel==Htn2LYRFRGGPONQ6Y7aNRw
TaskLabel==dtGSMvrkQ8aPgYfDEwoCrQ


jmaher: how do you tell if mach is generating the right taskgraph? Do the task graphs correspond to something that you can verify in the taskcluster tools?
Flags: needinfo?(jmaher)
Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

I also changed the worker type to m3.2xlarge but it seems that linux64-opt was allocated to it even though there are r3.2xlarge in the pool so the build wasn't faster. https://tools.taskcluster.net/aws-provisioner/#opt-linux64/
Attachment #8765924 - Flags: review?(gbrown)
:kmoir, this is all cryptic, but actually you solved it because when the patch that landed was there it was an error printed out while running |mach taskgraph target|, you don't see the error.

I have confidence all is well!
Flags: needinfo?(jmaher)
Comment on attachment 8765924 [details] [diff] [review]
bug1174206-2.patch

Review of attachment 8765924 [details] [diff] [review]:
-----------------------------------------------------------------

You may have it right already, but in light of bug 1283091, maybe double-check the x86 route.
Attachment #8765924 - Flags: review?(gbrown) → review+
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cee0d296a34
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89d45ec9d5aa
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown DONTBUILD
I changed the workertype back to android15 as it had a better pool of high memory instance types
https://hg.mozilla.org/mozilla-central/rev/1cee0d296a34
https://hg.mozilla.org/mozilla-central/rev/89d45ec9d5aa
Status: REOPENED → RESOLVED
Closed: 7 years ago6 years ago
Resolution: --- → FIXED
I have to add another patch to enable on other branches, this was just on try.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
patch to enable on other branches
Attachment #8766768 - Flags: review?(gbrown)
Attachment #8766768 - Flags: review?(gbrown) → review+
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4fc457de3a7
Migrate Android x86 builds to TaskCluster and make Tier 2 r=gbrown
This is landed on inbound and I've verified on treeherder However, I'm going to ask in the next mobile meeting if Android x86 is still needed. I recall asking in a meeting about six months ago that Android x86 has very small market numbers compared to armv7. Going to confirm if this is still the case and what the future holds for x86 builds from a product standpoint.
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
Blocks: 1293730
Duplicate of this bug: 1282197
Depends on: 1323485
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.