Closed
Bug 1405413
Opened 8 years ago
Closed 8 years ago
Migrate Gradle Deps task to toolchain dependencies
Categories
(Taskcluster :: General, enhancement)
Taskcluster
General
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla58
People
(Reporter: nalexander, Assigned: nalexander)
References
(Blocks 1 open bug)
Details
Attachments
(3 files, 1 obsolete file)
This ticket tracks turning the Gradle dependency fetching bits of the existing Deps task into a proper toolchain task.
I think this will be tricky 'cuz the existing Deps task is fundamentally "build-like": it configures and builds a tree (with !COMPILE_ENVIRONMENT) before running some Gradle tasks. That doesn't fit that well with the existing toolchain machinery, which expects to run a script in a non-build environment. But let's see how we do!
Assignee | ||
Updated•8 years ago
|
Blocks: gradle-automation-v2
Assignee | ||
Comment 1•8 years ago
|
||
Assignee | ||
Comment 2•8 years ago
|
||
The jobs at https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd13a02b190be61489106f1a38a26c9882dc7a0a are close to working, but:
[task 2017-10-03T22:34:08.776Z] 22:34:08 INFO - 0:06.30 403 Client Error: Forbidden for url: https://queue.taskcluster.net/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz
dustin: that must mean our business with the TC scopes didn't work. Can you take a look at that hacky patch series and suggest where I might have gone wrong?
Flags: needinfo?(dustin)
Assignee | ||
Comment 3•8 years ago
|
||
glandium suggested using the Task Cluster proxy in the relevant mach command, but a try push with that still gives
[task 2017-10-03T22:58:52.078Z] 22:58:52 INFO - 0:04.54 403 Client Error: Forbidden for url: http://taskcluster/queue/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz
Maybe my paths are wrong?
Comment 4•8 years ago
|
||
I'll be away for a while - maybe bstack can help?
Flags: needinfo?(dustin) → needinfo?(bstack)
Comment 5•8 years ago
|
||
Was this sorted out by the investigation from yesterday or is it still an issue?
Flags: needinfo?(bstack)
Assignee | ||
Comment 6•8 years ago
|
||
(In reply to Brian Stack [:bstack] from comment #5)
> Was this sorted out by the investigation from yesterday or is it still an
> issue?
I _think_ I've figured out: https://treeherder.mozilla.org/#/jobs?repo=try&revision=59ac3c431b3c6e33e4b49fba7c1a827c0982ec97&selectedJob=135245649 is downloading android-sdk-linux... which is private! I did have to work around the issue fixed by Bug 1405889 since that fix isn't deployed yet, but I was able to make the taskcluster-proxy /bewit endpoint work \o/
Thanks for checking in, bstack.
Assignee | ||
Comment 7•8 years ago
|
||
Assignee | ||
Comment 8•8 years ago
|
||
Assignee | ||
Comment 9•8 years ago
|
||
Assignee | ||
Comment 10•8 years ago
|
||
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 13•8 years ago
|
||
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
Sorry, jumped the gun -- this isn't quite ready yet!
Attachment #8916140 -
Flags: review?(dustin)
Comment 14•8 years ago
|
||
mozreview-review |
Comment on attachment 8916139 [details]
Bug 1405413 - Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies.
https://reviewboard.mozilla.org/r/187388/#review192822
"work through" instead of "work around"? :)
Attachment #8916139 -
Flags: review?(dustin) → review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 17•8 years ago
|
||
mozreview-review |
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
https://reviewboard.mozilla.org/r/187390/#review194248
::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16
(Diff revision 2)
> -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh
> +# We need a full checkout, and arranging a new sparse profile appears
> +# fraught with cache errors.
> +hg debugsparse --reset
dustin: this work is basically all done and ready for review... except for this. I had deep caching issues if I tried to run a toolchain task with a non-toolchain sparseprofile. This works, but might damage caches (I really don't know); I'm going to get glandium and gps to comment as well.
Assignee | ||
Comment 18•8 years ago
|
||
(In reply to Nick Alexander :nalexander from comment #17)
> Comment on attachment 8916140 [details]
> Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
>
> https://reviewboard.mozilla.org/r/187390/#review194248
>
> ::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16
> (Diff revision 2)
> > -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh
> > +# We need a full checkout, and arranging a new sparse profile appears
> > +# fraught with cache errors.
> > +hg debugsparse --reset
>
> dustin: this work is basically all done and ready for review... except for
> this. I had deep caching issues if I tried to run a toolchain task with a
> non-toolchain sparseprofile. This works, but might damage caches (I really
> don't know); I'm going to get glandium and gps to comment as well.
gps: glandium: I think you two are the most knowledgeable about the sparseprofile stuff. As I said, I tried to let the toolchain task override the sparseprofile set at http://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/toolchain.py#152, but I had very difficult to understand errors: missing files that were actually there; corrupt checkouts galore; etc. I found this as a work-around but want to get your opinion on this hack, and how we should address this in general. That is, what do we do for toolchain tasks that require larger and larger subsets of the source checkout?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
Comment 19•8 years ago
|
||
That's a good question, and I don't have an answer to it :( I don't know how much flexibility there is in the underlying mercurial feature.
Flags: needinfo?(mh+mozilla)
Comment 20•8 years ago
|
||
mozreview-review |
Comment on attachment 8916140 [details]
Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies.
https://reviewboard.mozilla.org/r/187390/#review194436
I know nothing about sparse checkouts, so I'm no use to you there :(
Attachment #8916140 -
Flags: review?(dustin) → review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 25•8 years ago
|
||
mozreview-review |
Comment on attachment 8918434 [details]
Bug 1405413 - Pre: Allow toolchain tasks to (un)set sparse-profile.
https://reviewboard.mozilla.org/r/189262/#review194612
Attachment #8918434 -
Flags: review+
Comment 26•8 years ago
|
||
mozreview-review |
Comment on attachment 8918435 [details]
Bug 1405413 - Don't use `hg debugsparse --reset`.
https://reviewboard.mozilla.org/r/189264/#review194614
Attachment #8918435 -
Flags: review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Updated•8 years ago
|
Attachment #8918435 -
Attachment is obsolete: true
Attachment #8918435 -
Flags: review?(dustin)
Comment 30•8 years ago
|
||
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cfb07a2fc671
Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies. r=dustin
https://hg.mozilla.org/integration/autoland/rev/5ee9c2e5e299
Pre: Allow toolchain tasks to (un)set sparse-profile. r=glandium
https://hg.mozilla.org/integration/autoland/rev/5ac2d6424a18
Migrate Gradle dependency fetching to toolchain dependencies. r=dustin
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
![]() |
||
Comment 31•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/cfb07a2fc671
https://hg.mozilla.org/mozilla-central/rev/5ee9c2e5e299
https://hg.mozilla.org/mozilla-central/rev/5ac2d6424a18
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Comment 32•8 years ago
|
||
We use a different set of caches depending on whether a sparse checkout is being used. And run-task + robustcheckout should have sufficient checks to ensure any accidental mismatch between sparse checkouts and caches is detected.
The sparse changes in this series LGTM.
Flags: needinfo?(gps)
You need to log in
before you can comment on or make changes to this bug.
Description
•