Closed Bug 1461980 Opened 2 years ago Closed 2 years ago

[run-task] Add ability to use artifacts from parent tasks

Categories

(Firefox Build System :: Task Configuration, task)

3 Branch
task
Not set

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(2 files)

When creating new test suites and tasks, I like to try and avoid mozharness/test packaging in favour of mach/running from the srcdir.

The latter approach is difficult to accomplish if the new task depends on any build artifacts. The mochitest/reftest selftest tasks do this, but it's very crude and not easy to copy over to other types of tasks. Basically all taskgraph does is set an environment variable with the url to the artifact, then there is code in the tasks themselves that will download/set it up.

I'd like a generic way for a task to say "I need these artifacts from this  dependency". Then I'd like the run-task script to automatically download (and possibly extract?) them into a known location.

I believe something like this would be a big step towards not needing mozharness anymore.
This might be a nice intermediate step toward gps's aspiration to have the worker do these downloads for you..
Ooh, that would be even better! But yeah, unless that proposal is imminent it's probably worthwhile having something like this in the meantime. I have some WIP patches going through try atm.
Blocks: 1445975
Comment on attachment 8977041 [details]
Bug 1461980 - [taskgraph] Add ability to download artifacts from dependencies in run-task script,

https://reviewboard.mozilla.org/r/244670/#review251516

::: taskcluster/taskgraph/transforms/job/run_task.py:34
(Diff revision 1)
>      # if true, perform a checkout of a comm-central based branch inside the
>      # gecko checkout
>      Required('comm-checkout'): bool,
>  
> +    # maps a dependency to a list of artifact names to use from that dependency.
> +    # E.g: {"build": ["target.tar.bz2"]}

It might be worth mentioning here where these will be downloaded.
Attachment #8977041 - Flags: review?(dustin) → review+
Comment on attachment 8977042 [details]
Bug 1461980 - [taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests,

https://reviewboard.mozilla.org/r/244672/#review251524
Attachment #8977042 - Flags: review?(dustin) → review+
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/aceeafcc22c9
[taskgraph] Add ability to download artifacts from dependencies in run-task script, r=dustin
https://hg.mozilla.org/integration/autoland/rev/bbeba6126822
[taskgraph] Use run-task's 'use-artifacts' key to setup artifacts in mochitest and reftest selftests, r=dustin
https://hg.mozilla.org/mozilla-central/rev/aceeafcc22c9
https://hg.mozilla.org/mozilla-central/rev/bbeba6126822
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Hah - I had kind of already written these patches. Although I never submitted them for review. That's on me.

The main difference from what I wrote is that I had support for validating the SHA-256 of artifacts. And I think I had support for extracting archives. We should be able to incorporate that into this scheme as follow-ups easily enough though.

Thanks for implementing this. It's a *really* useful feature. It will likely be replaced by features in the workers themselves. But until that day arrives, this bridges the gap quite nicely.
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.