Make Android Gradle dependency fetching TaskCluster task also fetch and package Android SDK

RESOLVED FIXED

Status

()

defect
RESOLVED FIXED
3 years ago
9 months ago

People

(Reporter: nalexander, Assigned: nalexander)

Tracking

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox56 fixed)

Details

Attachments

(4 attachments)

The dependency fetching task reworked in Bug 1252928 fetches Gradle and a host of Maven/Gradle dependencies, packages both up, and exposes them as TaskCluster artifacts.  From there they can be fetched and re-uploaded to tooltool for consumption by buildbot-backed builds.

The dependency fetching task is based on Jake Wharton's sdk-manager-plugin.  That plugin will happily download Android SDKs, extra dependencies, and even NDKs and emulators.  It would be great to make that task also handle fetching and re-packaging new Android SDK versions, etc.

There's some trickiness to the ordering here: |mach gradle| needs to have a valid configuration from |mach configure|; but |mach configure| verifies that valid Android SDKs, etc can be found.  We might need to make the Gradle configuration have a backdoor for when we want to use the sdk-manager-plugin.

Keep in mind that we can't redistribute these freely, so we might need private TaskCluster artifacts to make this happen.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 7

2 years ago
mozreview-review
Comment on attachment 8891011 [details]
Bug 1255227 - Pre: Run android-api-15-gradle-dependencies when underlying image changes.

https://reviewboard.mozilla.org/r/162192/#review167496
Attachment #8891011 - Flags: review?(dustin) → review+

Comment 8

2 years ago
mozreview-review
Comment on attachment 8891013 [details]
Bug 1255227 - Part 2: Bootstrap and upload android-sdk-linux.tar.xz.

https://reviewboard.mozilla.org/r/162196/#review167500

Just to check my understanding, the intent here is just to produce the SDK as an artifact which can be manually uploaded.  Will you return afterward to hook this up as a dependent task, so that downstream tasks can use the resulting SDK?  If that's the case, would it make sense to move the gradle-dependencies task into one or more tasks in the toolchain kind?  There's already support for downloading toolchain artifacts for builds (although it doesn't support private artifacts yet..)

::: mobile/android/config/mozconfigs/android-api-15-gradle-dependencies/nightly:48
(Diff revision 1)
>  export MOZILLA_OFFICIAL=1
>  export MOZ_TELEMETRY_REPORTING=1
>  
>  . "$topsrcdir/mobile/android/config/mozconfigs/common.override"
> +
> +# End ../android-api-15-frontend/nightly.

what is this line?
Attachment #8891013 - Flags: review?(dustin) → review+
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Comment on attachment 8891013 [details]
> Bug 1255227 - Part 2: Bootstrap and upload android-sdk-linux.tar.xz.
> 
> https://reviewboard.mozilla.org/r/162196/#review167500
> 
> Just to check my understanding, the intent here is just to produce the SDK
> as an artifact which can be manually uploaded.  Will you return afterward to
> hook this up as a dependent task, so that downstream tasks can use the
> resulting SDK?  If that's the case, would it make sense to move the
> gradle-dependencies task into one or more tasks in the toolchain kind? 
> There's already support for downloading toolchain artifacts for builds
> (although it doesn't support private artifacts yet..)

From IRC:

Re: converting Android dependencies to toolchain tasks -- absolutely, that's the plan.  Sadly I won't be able to get to it for quite a while (parental leave), but it's all structured to migrate that way.  I didn't push it for this round of changes (which are significant, and a bitch to test manually!) since it doesn't handle private artifacts yet.  And since it's undocumented!  But at least we're running the repackage script in a task now.  And running the Android bootstrap in a task too, so that we're not hand-rolling those things.

> :::
> mobile/android/config/mozconfigs/android-api-15-gradle-dependencies/nightly:
> 48
> (Diff revision 1)
> >  export MOZILLA_OFFICIAL=1
> >  export MOZ_TELEMETRY_REPORTING=1
> >  
> >  . "$topsrcdir/mobile/android/config/mozconfigs/common.override"
> > +
> > +# End ../android-api-15-frontend/nightly.
> 
> what is this line?

Oh, I noticed there was an opener ("just like ...") and this thing, which has to happen _after_ the other stuff to override, is now no longer like ".../nightly".  So I closed the opener to delimit the bits that should be the same.

Comment 10

2 years ago
mozreview-review
Comment on attachment 8891010 [details]
Bug 1255227 - Pre: Use {0} for Python 2.6.

https://reviewboard.mozilla.org/r/162190/#review167688
Attachment #8891010 - Flags: review?(s.kaspari) → review+

Comment 11

2 years ago
mozreview-review
Comment on attachment 8891012 [details]
Bug 1255227 - Part 1: Stop using deprecated android-sdk-manager Gradle plugin.

https://reviewboard.mozilla.org/r/162194/#review167690
Attachment #8891012 - Flags: review?(s.kaspari) → review+

Comment 12

2 years ago
mozreview-review
Comment on attachment 8891013 [details]
Bug 1255227 - Part 2: Bootstrap and upload android-sdk-linux.tar.xz.

https://reviewboard.mozilla.org/r/162196/#review167692

::: taskcluster/docker/android-gradle-build/bin/after.sh:18
(Diff revision 1)
> +cp -R /home/worker/.mozbuild/android-sdk-linux android-sdk-linux
> +tar cJf android-sdk-linux.tar.xz android-sdk-linux
> +
> +# We can't redistribute the Android SDK publicly.
> +mkdir -p /home/worker/private/android-sdk
> +mv android-sdk-linux.tar.xz /home/worker/private/android-sdk

I assume for now I'd still need to upload this to tooltool myself? Is there a way for me to access this private file?
Attachment #8891013 - Flags: review?(s.kaspari) → review+
(In reply to Sebastian Kaspari (:sebastian) from comment #12)
> Comment on attachment 8891013 [details]
> Bug 1255227 - Part 2: Bootstrap and upload android-sdk-linux.tar.xz.
> 
> https://reviewboard.mozilla.org/r/162196/#review167692
> 
> ::: taskcluster/docker/android-gradle-build/bin/after.sh:18
> (Diff revision 1)
> > +cp -R /home/worker/.mozbuild/android-sdk-linux android-sdk-linux
> > +tar cJf android-sdk-linux.tar.xz android-sdk-linux
> > +
> > +# We can't redistribute the Android SDK publicly.
> > +mkdir -p /home/worker/private/android-sdk
> > +mv android-sdk-linux.tar.xz /home/worker/private/android-sdk
> 
> I assume for now I'd still need to upload this to tooltool myself?

Correct.  I'll file tickets for turning these into new-style toolchain-* tasks shortly, which should remove this manual step.  (Eventually, if and when we support private toolchain artifacts.)

 Is there
> a way for me to access this private file?

Yes!  The Tree Herder links don't work (another ticket to file), but the Task Cluster links do.  Choose the Task Cluster task in the bottom-left pane, then choose "Run Artifacts" (the far right option), and then choose the artifact.  The link will be identical but your signed in state (to Task Cluster itself) will be used and you'll be able to download.  Then you can push to tooltool with the "internal" visibility, like usual.  All MoCo employees have read access to private/*.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 22

2 years ago
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/547273e23ee4
Pre: Use {0} for Python 2.6. r=sebastian
https://hg.mozilla.org/integration/autoland/rev/3878c778174c
Pre: Run android-api-15-gradle-dependencies when underlying image changes. r=dustin
https://hg.mozilla.org/integration/autoland/rev/5b150868209b
Part 1: Stop using deprecated android-sdk-manager Gradle plugin. r=sebastian
https://hg.mozilla.org/integration/autoland/rev/23330f35f957
Part 2: Bootstrap and upload android-sdk-linux.tar.xz. r=dustin,sebastian
Depends on: 1390824
No longer depends on: 1390824
Component: Build Config → Build Config & IDE Support
Product: Core → Firefox for Android
Target Milestone: mozilla56 → ---
Assignee: nobody → nalexander
You need to log in before you can comment on or make changes to this bug.