Closed
Bug 1405413
Opened 5 years ago
Closed 5 years ago
Migrate Gradle Deps task to toolchain dependencies
Categories
(Taskcluster :: General, enhancement)
Taskcluster
General
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla58
People
(Reporter: nalexander, Assigned: nalexander)
References
(Blocks 1 open bug)
Details
Attachments
(3 files, 1 obsolete file)
This ticket tracks turning the Gradle dependency fetching bits of the existing Deps task into a proper toolchain task. I think this will be tricky 'cuz the existing Deps task is fundamentally "build-like": it configures and builds a tree (with !COMPILE_ENVIRONMENT) before running some Gradle tasks. That doesn't fit that well with the existing toolchain machinery, which expects to run a script in a non-build environment. But let's see how we do!
Assignee | ||
Updated•5 years ago
|
Blocks: gradle-automation-v2
Assignee | ||
Comment 1•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4924eb0c87cce0ddbddad7ed67573f62c65c3ab
Assignee | ||
Comment 2•5 years ago
|
||
The jobs at https://treeherder.mozilla.org/#/jobs?repo=try&revision=cd13a02b190be61489106f1a38a26c9882dc7a0a are close to working, but: [task 2017-10-03T22:34:08.776Z] 22:34:08 INFO - 0:06.30 403 Client Error: Forbidden for url: https://queue.taskcluster.net/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz dustin: that must mean our business with the TC scopes didn't work. Can you take a look at that hacky patch series and suggest where I might have gone wrong?
Flags: needinfo?(dustin)
Assignee | ||
Comment 3•5 years ago
|
||
glandium suggested using the Task Cluster proxy in the relevant mach command, but a try push with that still gives [task 2017-10-03T22:58:52.078Z] 22:58:52 INFO - 0:04.54 403 Client Error: Forbidden for url: http://taskcluster/queue/v1/task/V8wk0GL4TaOfTeyNoSRlyQ/artifacts/project/gecko/android-sdk/android-sdk-linux.tar.xz Maybe my paths are wrong?
Comment 4•5 years ago
|
||
I'll be away for a while - maybe bstack can help?
Flags: needinfo?(dustin) → needinfo?(bstack)
Comment 5•5 years ago
|
||
Was this sorted out by the investigation from yesterday or is it still an issue?
Flags: needinfo?(bstack)
Assignee | ||
Comment 6•5 years ago
|
||
(In reply to Brian Stack [:bstack] from comment #5) > Was this sorted out by the investigation from yesterday or is it still an > issue? I _think_ I've figured out: https://treeherder.mozilla.org/#/jobs?repo=try&revision=59ac3c431b3c6e33e4b49fba7c1a827c0982ec97&selectedJob=135245649 is downloading android-sdk-linux... which is private! I did have to work around the issue fixed by Bug 1405889 since that fix isn't deployed yet, but I was able to make the taskcluster-proxy /bewit endpoint work \o/ Thanks for checking in, bstack.
Assignee | ||
Comment 7•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2149281ad2a1cde7694f9c1e276a5215291bd7f8
Assignee | ||
Comment 8•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cff40ecbfefb9c72441abcce620a51153c05e8a
Assignee | ||
Comment 9•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4b4d09a45e18a7e17eb62229c6d33c3d2fe59c7
Assignee | ||
Comment 10•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a2e90d63322913e8634361eb9f49d60682732f30
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 13•5 years ago
|
||
Comment on attachment 8916140 [details] Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies. Sorry, jumped the gun -- this isn't quite ready yet!
Attachment #8916140 -
Flags: review?(dustin)
Comment 14•5 years ago
|
||
mozreview-review |
Comment on attachment 8916139 [details] Bug 1405413 - Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies. https://reviewboard.mozilla.org/r/187388/#review192822 "work through" instead of "work around"? :)
Attachment #8916139 -
Flags: review?(dustin) → review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 17•5 years ago
|
||
mozreview-review |
Comment on attachment 8916140 [details] Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies. https://reviewboard.mozilla.org/r/187390/#review194248 ::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16 (Diff revision 2) > -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh > +# We need a full checkout, and arranging a new sparse profile appears > +# fraught with cache errors. > +hg debugsparse --reset dustin: this work is basically all done and ready for review... except for this. I had deep caching issues if I tried to run a toolchain task with a non-toolchain sparseprofile. This works, but might damage caches (I really don't know); I'm going to get glandium and gps to comment as well.
Assignee | ||
Comment 18•5 years ago
|
||
(In reply to Nick Alexander :nalexander from comment #17) > Comment on attachment 8916140 [details] > Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies. > > https://reviewboard.mozilla.org/r/187390/#review194248 > > ::: taskcluster/scripts/misc/android-gradle-dependencies.sh:16 > (Diff revision 2) > > -. $WORKSPACE/build/src/taskcluster/scripts/builder/build-android-dependencies/after.sh > > +# We need a full checkout, and arranging a new sparse profile appears > > +# fraught with cache errors. > > +hg debugsparse --reset > > dustin: this work is basically all done and ready for review... except for > this. I had deep caching issues if I tried to run a toolchain task with a > non-toolchain sparseprofile. This works, but might damage caches (I really > don't know); I'm going to get glandium and gps to comment as well. gps: glandium: I think you two are the most knowledgeable about the sparseprofile stuff. As I said, I tried to let the toolchain task override the sparseprofile set at http://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/toolchain.py#152, but I had very difficult to understand errors: missing files that were actually there; corrupt checkouts galore; etc. I found this as a work-around but want to get your opinion on this hack, and how we should address this in general. That is, what do we do for toolchain tasks that require larger and larger subsets of the source checkout?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
Comment 19•5 years ago
|
||
That's a good question, and I don't have an answer to it :( I don't know how much flexibility there is in the underlying mercurial feature.
Flags: needinfo?(mh+mozilla)
Comment 20•5 years ago
|
||
mozreview-review |
Comment on attachment 8916140 [details] Bug 1405413 - Migrate Gradle dependency fetching to toolchain dependencies. https://reviewboard.mozilla.org/r/187390/#review194436 I know nothing about sparse checkouts, so I'm no use to you there :(
Attachment #8916140 -
Flags: review?(dustin) → review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 25•5 years ago
|
||
mozreview-review |
Comment on attachment 8918434 [details] Bug 1405413 - Pre: Allow toolchain tasks to (un)set sparse-profile. https://reviewboard.mozilla.org/r/189262/#review194612
Attachment #8918434 -
Flags: review+
Comment 26•5 years ago
|
||
mozreview-review |
Comment on attachment 8918435 [details] Bug 1405413 - Don't use `hg debugsparse --reset`. https://reviewboard.mozilla.org/r/189264/#review194614
Attachment #8918435 -
Flags: review+
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Updated•5 years ago
|
Attachment #8918435 -
Attachment is obsolete: true
Attachment #8918435 -
Flags: review?(dustin)
Comment 30•5 years ago
|
||
Pushed by nalexander@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/cfb07a2fc671 Pre: Expand sparse checkouts in taskgraph for android-gradle-dependencies. r=dustin https://hg.mozilla.org/integration/autoland/rev/5ee9c2e5e299 Pre: Allow toolchain tasks to (un)set sparse-profile. r=glandium https://hg.mozilla.org/integration/autoland/rev/5ac2d6424a18 Migrate Gradle dependency fetching to toolchain dependencies. r=dustin
Assignee | ||
Updated•5 years ago
|
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
![]() |
||
Comment 31•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/cfb07a2fc671 https://hg.mozilla.org/mozilla-central/rev/5ee9c2e5e299 https://hg.mozilla.org/mozilla-central/rev/5ac2d6424a18
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Comment 32•5 years ago
|
||
We use a different set of caches depending on whether a sparse checkout is being used. And run-task + robustcheckout should have sufficient checks to ensure any accidental mismatch between sparse checkouts and caches is detected. The sparse changes in this series LGTM.
Flags: needinfo?(gps)
You need to log in
before you can comment on or make changes to this bug.
Description
•