Closed Bug 1405413 Opened 5 years ago Closed 5 years ago

Migrate Gradle Deps task to toolchain dependencies

Categories

(Taskcluster :: General, enhancement)

Product:
Taskcluster
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla58

People

(Reporter: nalexander, Assigned: nalexander)

References

(Blocks 1 open bug)

Details

Attachments

(3 files, 1 obsolete file)

Bug 1405413 - Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405413 - Pre: Allow toolchain tasks to (un)set sparse-profile.
59 bytes, text/x-review-board-request
glandium
: review+
Details 
This ticket tracks turning the Gradle dependency fetching bits of the existing Deps task into a proper toolchain task.

I think this will be tricky 'cuz the existing Deps task is fundamentally "build-like": it configures and builds a tree (with !COMPILE_ENVIRONMENT) before running some Gradle tasks.  That doesn't fit that well with the existing toolchain machinery, which expects to run a script in a non-build environment.  But let's see how we do!
The jobs at https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd13a02b190be61489106f1a38a26c9882dc7a0a are close to working, but:

[task 2017-10-03T22:34:08.776Z] 22:34:08     INFO -   0:06.30 403 Client Error: Forbidden for url: https://queue.taskcluster.net/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz

dustin: that must mean our business with the TC scopes didn't work.  Can you take a look at that hacky patch series and suggest where I might have gone wrong?
Flags: needinfo?(dustin)
glandium suggested using the Task Cluster proxy in the relevant mach command, but a try push with that still gives

[task 2017-10-03T22:58:52.078Z] 22:58:52     INFO -   0:04.54 403 Client Error: Forbidden for url: http://taskcluster/queue/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz

Maybe my paths are wrong?
I'll be away for a while - maybe bstack can help?
Flags: needinfo?(dustin) → needinfo?(bstack)
Was this sorted out by the investigation from yesterday or is it still an issue?
Flags: needinfo?(bstack)
(In reply to Brian Stack [:bstack] from comment #5)
> Was this sorted out by the investigation from yesterday or is it still an
> issue?

I _think_ I've figured out: https://treeherder.mozilla.org/#/jobs?repo=try&revision=59ac3c431b3c6e33e4b49fba7c1a827c0982ec97&selectedJob=135245649 is downloading android-sdk-linux... which is private!  I did have to work around the issue fixed by Bug 1405889 since that fix isn't deployed yet, but I was able to make the taskcluster-proxy /bewit endpoint work \o/

Thanks for checking in, bstack.
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.

Sorry, jumped the gun -- this isn't quite ready yet!
Attachment #8916140 - Flags: review?(dustin)
Comment on attachment 8916139 [details]
Bug 1405413 - Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies.

https://reviewboard.mozilla.org/r/187388/#review192822

"work through" instead of "work around"? :)
Attachment #8916139 - Flags: review?(dustin) → review+
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.

https://reviewboard.mozilla.org/r/187390/#review194248

::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16
(Diff revision 2)
> -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh
> +# We need a full checkout, and arranging a new sparse profile appears
> +# fraught with cache errors.
> +hg debugsparse --reset

dustin: this work is basically all done and ready for review... except for this.  I had deep caching issues if I tried to run a toolchain task with a non-toolchain sparseprofile.  This works, but might damage caches (I really don't know); I'm going to get glandium and gps to comment as well.
(In reply to Nick Alexander :nalexander from comment #17)
> Comment on attachment 8916140 [details]
> Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
> 
> https://reviewboard.mozilla.org/r/187390/#review194248
> 
> ::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16
> (Diff revision 2)
> > -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh
> > +# We need a full checkout, and arranging a new sparse profile appears
> > +# fraught with cache errors.
> > +hg debugsparse --reset
> 
> dustin: this work is basically all done and ready for review... except for
> this.  I had deep caching issues if I tried to run a toolchain task with a
> non-toolchain sparseprofile.  This works, but might damage caches (I really
> don't know); I'm going to get glandium and gps to comment as well.

gps: glandium: I think you two are the most knowledgeable about the sparseprofile stuff.  As I said, I tried to let the toolchain task override the sparseprofile set at http://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/toolchain.py#152, but I had very difficult to understand errors: missing files that were actually there; corrupt checkouts galore; etc.  I found this as a work-around but want to get your opinion on this hack, and how we should address this in general.  That is, what do we do for toolchain tasks that require larger and larger subsets of the source checkout?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
That's a good question, and I don't have an answer to it :( I don't know how much flexibility there is in the underlying mercurial feature.
Flags: needinfo?(mh+mozilla)
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.

https://reviewboard.mozilla.org/r/187390/#review194436

I know nothing about sparse checkouts, so I'm no use to you there :(
Attachment #8916140 - Flags: review?(dustin) → review+
Comment on attachment 8918434 [details]
Bug 1405413 - Pre: Allow toolchain tasks to (un)set sparse-profile.

https://reviewboard.mozilla.org/r/189262/#review194612
Attachment #8918434 - Flags: review+
Comment on attachment 8918435 [details]
Bug 1405413 - Don't use `hg debugsparse --reset`.

https://reviewboard.mozilla.org/r/189264/#review194614
Attachment #8918435 - Flags: review+
Attachment #8918435 - Attachment is obsolete: true
Attachment #8918435 - Flags: review?(dustin)
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cfb07a2fc671
Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies. r=dustin
https://hg.mozilla.org/integration/autoland/rev/5ee9c2e5e299
Pre: Allow toolchain tasks to (un)set sparse-profile. r=glandium
https://hg.mozilla.org/integration/autoland/rev/5ac2d6424a18
Migrate Gradle dependency fetching to toolchain dependencies. r=dustin
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
We use a different set of caches depending on whether a sparse checkout is being used. And run-task + robustcheckout should have sufficient checks to ensure any accidental mismatch between sparse checkouts and caches is detected.

The sparse changes in this series LGTM.
Flags: needinfo?(gps)
Blocks: 1429210
You need to log in before you can comment on or make changes to this bug.